[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:43 dist-test-slave-8vw7 krb5kdc[9104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:43 dist-test-slave-8vw7 krb5kdc[9104](info): set up 2 sockets
Sep 25 15:18:43 dist-test-slave-8vw7 krb5kdc[9104](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:18:44.285286 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:18:44 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:18:44.441373 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:36971
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:44.741809 9117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:44.742067 9117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:44.742240 9117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:44.748863 9117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:44.748961 9117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:44.749029 9117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:44.749109 9117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:44.749188 9117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:44.758127 9117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:44.759383 9117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:44.760640 9117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:44.776824 9123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:44.776847 9122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:44.777244 9125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:44.777653 9117 server_base.cc:1034] running on GCE node
I20260925 15:18:44.778306 9117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:44.779527 9117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:44.780714 9117 hybrid_clock.cc:648] HybridClock initialized: now 1790349524780656 us; error 52 us; skew 500 ppm
I20260925 15:18:44.781081 9117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:44 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349524, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:44.784096 9117 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:18:44.785610 9117 webserver.cc:460] Webserver started at http://127.8.226.126:43375/ using document root <none> and password file <none>
I20260925 15:18:44.786406 9117 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:44.786551 9117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:44.786898 9117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:44.789834 9117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96"
format_stamp: "Formatted at 2026-09-25 15:18:44 on dist-test-slave-8vw7"
server_key: "d7ca8b91aa24dcae3172a54348dacb79"
server_key_iv: "301cab2b324bc23d513799fd0d6d5912"
server_key_version: "encryptionkey@0"
I20260925 15:18:44.790634 9117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96"
format_stamp: "Formatted at 2026-09-25 15:18:44 on dist-test-slave-8vw7"
server_key: "d7ca8b91aa24dcae3172a54348dacb79"
server_key_iv: "301cab2b324bc23d513799fd0d6d5912"
server_key_version: "encryptionkey@0"
I20260925 15:18:44.795557 9117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:18:44.799057 9132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:44.800266 9117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:44.800427 9117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96"
format_stamp: "Formatted at 2026-09-25 15:18:44 on dist-test-slave-8vw7"
server_key: "d7ca8b91aa24dcae3172a54348dacb79"
server_key_iv: "301cab2b324bc23d513799fd0d6d5912"
server_key_version: "encryptionkey@0"
I20260925 15:18:44.800597 9117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:44.828493 9117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:44.832739 9117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:44.833024 9117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:44.850732 9117 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:36971
I20260925 15:18:44.850760 9183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:36971 every 8 connection(s)
I20260925 15:18:44.852025 9117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:44.855415 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9117
I20260925 15:18:44.855757 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:18:44.856082 9184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:44.858186 9097 external_mini_cluster.cc:1428] Setting key fde0a1bb800ef6841b588f6962f0e153
I20260925 15:18:44.868039 9184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96: Bootstrap starting.
I20260925 15:18:44.870814 9184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:44.871675 9184 log.cc:824] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:44.874260 9184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96: No bootstrap required, opened a new log
I20260925 15:18:44.883522 9184 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36971 } }
I20260925 15:18:44.883898 9184 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:44.883993 9184 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0191fb6a53247a7bf1f8fd02d2e5f96, State: Initialized, Role: FOLLOWER
I20260925 15:18:44.884692 9184 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36971 } }
I20260925 15:18:44.885010 9184 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:44.885198 9184 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:44.885397 9184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:44.887435 9184 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36971 } }
I20260925 15:18:44.887858 9184 leader_election.cc:302] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0191fb6a53247a7bf1f8fd02d2e5f96; no voters:
I20260925 15:18:44.888437 9184 leader_election.cc:288] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:44.888574 9189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:44.889807 9189 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [term 1 LEADER]: Becoming Leader. State: Replica: d0191fb6a53247a7bf1f8fd02d2e5f96, State: Running, Role: LEADER
Sep 25 15:18:44 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:44.890446 9184 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:44.890317 9189 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36971 } }
I20260925 15:18:44.893361 9190 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36971 } } }
I20260925 15:18:44.893748 9190 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:44.897725 9191 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0191fb6a53247a7bf1f8fd02d2e5f96. Latest consensus state: current_term: 1 leader_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0191fb6a53247a7bf1f8fd02d2e5f96" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36971 } } }
I20260925 15:18:44.898063 9191 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:44.906219 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:44.863257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.8.226.126:36971)
0925 15:18:44.863910 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:44.863921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:44.867573 (+ 3652us) server_negotiation.cc:407] Connection header received
0925 15:18:44.876352 (+ 8779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:44.876375 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:44.876868 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:44.877276 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:44.879918 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:44.882073 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:44.883678 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:44.884649 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:44.884879 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:44.891431 (+ 6552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:44.891442 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:44.891484 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:44.891557 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:44.897004 (+ 5447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:44.898411 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:44.898424 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:44.898434 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:44.898741 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:44.899465 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:44.899472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:44.899477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:44.900285 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:44.900639 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:44.904158 (+ 3519us) server_negotiation.cc:299] Negotiation successful
0925 15:18:44.904617 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":292,"threads_started":1}
I20260925 15:18:44.911072 9196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:44.915925 9196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:44.924014 9196 catalog_manager.cc:1347] Generated new cluster ID: 7597a0d66f7049128f8c9bb3e697eea2
I20260925 15:18:44.924103 9196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:44.972579 9196 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:44.973841 9196 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:44.985337 9196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0191fb6a53247a7bf1f8fd02d2e5f96: Generated new TSK 0
I20260925 15:18:44.986176 9196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:18:45.036144 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36971
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:45.342618 9210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:45.342872 9210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:45.343047 9210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:45.349846 9210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:45.349937 9210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:45.350152 9210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:18:45.358585 9210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36971
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:45.359778 9210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:45.361100 9210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:45.373183 9218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:45.373354 9216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:45.373322 9215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:45.374185 9210 server_base.cc:1034] running on GCE node
I20260925 15:18:45.374719 9210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:45.375576 9210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:45.376762 9210 hybrid_clock.cc:648] HybridClock initialized: now 1790349525376720 us; error 30 us; skew 500 ppm
I20260925 15:18:45.377095 9210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:45 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349525, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:45.380107 9210 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:18:45.381438 9210 webserver.cc:460] Webserver started at http://127.8.226.65:36097/ using document root <none> and password file <none>
I20260925 15:18:45.382028 9210 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:45.382156 9210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:45.382434 9210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:45.385161 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "4661fa43ae9b46c5a7f2185e1f2c3185"
format_stamp: "Formatted at 2026-09-25 15:18:45 on dist-test-slave-8vw7"
server_key: "2219c9d458226b52f6f0798d688f09c4"
server_key_iv: "d3898d2e1befb37494fc670ff9adb49a"
server_key_version: "encryptionkey@0"
I20260925 15:18:45.385927 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "4661fa43ae9b46c5a7f2185e1f2c3185"
format_stamp: "Formatted at 2026-09-25 15:18:45 on dist-test-slave-8vw7"
server_key: "2219c9d458226b52f6f0798d688f09c4"
server_key_iv: "d3898d2e1befb37494fc670ff9adb49a"
server_key_version: "encryptionkey@0"
I20260925 15:18:45.390497 9210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:18:45.393551 9225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:45.394734 9210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:18:45.394896 9210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "4661fa43ae9b46c5a7f2185e1f2c3185"
format_stamp: "Formatted at 2026-09-25 15:18:45 on dist-test-slave-8vw7"
server_key: "2219c9d458226b52f6f0798d688f09c4"
server_key_iv: "d3898d2e1befb37494fc670ff9adb49a"
server_key_version: "encryptionkey@0"
I20260925 15:18:45.395076 9210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:45.436002 9210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:45.439764 9210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:45.440042 9210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:45.440891 9210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:45.442072 9210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:45.442166 9210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:45.442297 9210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:45.442373 9210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:45.474085 9210 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:33981
I20260925 15:18:45.474159 9337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:33981 every 8 connection(s)
I20260925 15:18:45.475260 9210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:45.482888 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9210
I20260925 15:18:45.483098 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:18:45.483834 9097 external_mini_cluster.cc:1428] Setting key 0833e3fe72084178dcda53a742a523ee
Sep 25 15:18:45 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349525, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:45.496392 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:45.478087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:43483 (local address 127.8.226.126:36971)
0925 15:18:45.478507 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:45.478513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:45.479410 (+ 897us) server_negotiation.cc:407] Connection header received
0925 15:18:45.481015 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:45.481019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:45.481078 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:45.481166 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:45.482453 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.483273 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.485260 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.485958 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.486052 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:45.490254 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:45.490262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:45.490266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:45.490307 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:45.493382 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:45.493987 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:45.493994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:45.494000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:45.494132 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:45.494480 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:45.494487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:45.494492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:45.494834 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:45.494966 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:45.495648 (+ 682us) server_negotiation.cc:299] Negotiation successful
0925 15:18:45.495920 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260925 15:18:45.496927 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:45.478627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36971 (local address 127.8.226.65:43483)
0925 15:18:45.479245 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:45.479289 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:45.480756 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:45.481343 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:45.481350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:45.481797 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:45.482317 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:45.482331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.483424 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.483430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:45.485094 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:45.485128 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.486163 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.486172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:45.486455 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:45.487161 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:45.487192 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:45.490019 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:45.493496 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:45.493502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:45.493512 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:45.493879 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:45.494228 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:45.494231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:45.494234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:45.494382 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:45.494943 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:45.494948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:45.495200 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 15:18:45.495523 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 15:18:45.495882 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":246,"threads_started":1}
I20260925 15:18:45.498431 9338 heartbeater.cc:347] Connected to a master server at 127.8.226.126:36971
I20260925 15:18:45.498723 9338 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:45.499394 9338 heartbeater.cc:511] Master 127.8.226.126:36971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:18:45.501349 9149 ts_manager.cc:194] Registered new tserver with Master: 4661fa43ae9b46c5a7f2185e1f2c3185 (127.8.226.65:33981)
I20260925 15:18:45.502668 9149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:43483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:18:45.520167 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36971
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:45.818966 9344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:45.819273 9344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:45.819437 9344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:45.826897 9344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:45.827013 9344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:45.827220 9344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:18:45.836593 9344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36971
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:45.837847 9344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:45.839102 9344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:45.851471 9350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:45.851426 9349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:45.852063 9352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:45.852203 9344 server_base.cc:1034] running on GCE node
I20260925 15:18:45.852735 9344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:45.853714 9344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:45.854923 9344 hybrid_clock.cc:648] HybridClock initialized: now 1790349525854883 us; error 35 us; skew 500 ppm
I20260925 15:18:45.855201 9344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:45 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349525, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:45.858228 9344 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:18:45.859622 9344 webserver.cc:460] Webserver started at http://127.8.226.66:41837/ using document root <none> and password file <none>
I20260925 15:18:45.860215 9344 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:45.860332 9344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:45.860615 9344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:45.863349 9344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "db584ebf0d344c9bb06102216c3cbd9c"
format_stamp: "Formatted at 2026-09-25 15:18:45 on dist-test-slave-8vw7"
server_key: "32f0cdf8e31a3e6ec21c941b763d4f38"
server_key_iv: "fe127de2ddaff8db1ac402c26036b796"
server_key_version: "encryptionkey@0"
I20260925 15:18:45.864049 9344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "db584ebf0d344c9bb06102216c3cbd9c"
format_stamp: "Formatted at 2026-09-25 15:18:45 on dist-test-slave-8vw7"
server_key: "32f0cdf8e31a3e6ec21c941b763d4f38"
server_key_iv: "fe127de2ddaff8db1ac402c26036b796"
server_key_version: "encryptionkey@0"
I20260925 15:18:45.868619 9344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:18:45.871824 9359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:45.873449 9344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:45.873613 9344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "db584ebf0d344c9bb06102216c3cbd9c"
format_stamp: "Formatted at 2026-09-25 15:18:45 on dist-test-slave-8vw7"
server_key: "32f0cdf8e31a3e6ec21c941b763d4f38"
server_key_iv: "fe127de2ddaff8db1ac402c26036b796"
server_key_version: "encryptionkey@0"
I20260925 15:18:45.873780 9344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:45.894634 9344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:45.898427 9344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:45.898676 9344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:45.899596 9344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:45.900722 9344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:45.900818 9344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:45.900951 9344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:45.901033 9344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:45.930642 9344 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:42383
I20260925 15:18:45.930702 9471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:42383 every 8 connection(s)
I20260925 15:18:45.931862 9344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:45.933362 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9344
I20260925 15:18:45.933553 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:18:45.934084 9097 external_mini_cluster.cc:1428] Setting key 18dae7d2c9301444e836be315c176512
Sep 25 15:18:45 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349525, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:18:45.951182 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:45.934356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:60653 (local address 127.8.226.126:36971)
0925 15:18:45.934515 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:45.934519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:45.935968 (+ 1449us) server_negotiation.cc:407] Connection header received
0925 15:18:45.937949 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:45.937953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:45.938030 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:45.938135 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:45.939475 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.940164 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.941586 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.941956 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.942078 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:45.945074 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:45.945082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:45.945089 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:45.945129 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:45.948496 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:45.949212 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:45.949216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:45.949218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:45.949343 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:45.949732 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:45.949737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:45.949740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:45.950000 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:45.950115 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:45.950737 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 15:18:45.950945 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
I20260925 15:18:45.952070 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:45.934966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36971 (local address 127.8.226.66:60653)
0925 15:18:45.935809 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:45.935859 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:45.937706 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:45.938370 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:45.938377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:45.938821 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:45.939325 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:45.939341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.940311 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.940315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:45.941459 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:45.941469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:45.942151 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:45.942159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:45.942364 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:45.942964 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:45.942983 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:45.944848 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:45.948647 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:45.948657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:45.948673 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:45.949111 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:45.949435 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:45.949438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:45.949440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:45.949602 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:45.950109 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:45.950115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:45.950383 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 15:18:45.950594 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 15:18:45.950989 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260925 15:18:45.953305 9472 heartbeater.cc:347] Connected to a master server at 127.8.226.126:36971
I20260925 15:18:45.953590 9472 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:45.954314 9472 heartbeater.cc:511] Master 127.8.226.126:36971 requested a full tablet report, sending...
I20260925 15:18:45.955881 9149 ts_manager.cc:194] Registered new tserver with Master: db584ebf0d344c9bb06102216c3cbd9c (127.8.226.66:42383)
I20260925 15:18:45.956643 9149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:60653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:18:45.970391 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36971
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:46.274250 9477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:46.274483 9477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:46.274566 9477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:46.281524 9477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:46.281625 9477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:46.281756 9477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:18:46.291069 9477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36971
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:46.292300 9477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:46.293550 9477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:46.305817 9483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:46.305841 9485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:46.305781 9482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:46.306579 9477 server_base.cc:1034] running on GCE node
I20260925 15:18:46.307073 9477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:46.307897 9477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:46.309039 9477 hybrid_clock.cc:648] HybridClock initialized: now 1790349526308989 us; error 47 us; skew 500 ppm
I20260925 15:18:46.309338 9477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:46.312460 9477 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:18:46.313762 9477 webserver.cc:460] Webserver started at http://127.8.226.67:37385/ using document root <none> and password file <none>
I20260925 15:18:46.314397 9477 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:46.314522 9477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:46.314821 9477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:46.317553 9477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "8c4571b089c5438aba7f81f6f76dc169"
format_stamp: "Formatted at 2026-09-25 15:18:46 on dist-test-slave-8vw7"
server_key: "3c55bef0c9c3fe5b932b0712ad2dfa5f"
server_key_iv: "4b855d0c665e06bb5a8611a576568816"
server_key_version: "encryptionkey@0"
I20260925 15:18:46.318276 9477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "8c4571b089c5438aba7f81f6f76dc169"
format_stamp: "Formatted at 2026-09-25 15:18:46 on dist-test-slave-8vw7"
server_key: "3c55bef0c9c3fe5b932b0712ad2dfa5f"
server_key_iv: "4b855d0c665e06bb5a8611a576568816"
server_key_version: "encryptionkey@0"
I20260925 15:18:46.323064 9477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:18:46.326335 9492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:46.327587 9477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:18:46.327766 9477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "8c4571b089c5438aba7f81f6f76dc169"
format_stamp: "Formatted at 2026-09-25 15:18:46 on dist-test-slave-8vw7"
server_key: "3c55bef0c9c3fe5b932b0712ad2dfa5f"
server_key_iv: "4b855d0c665e06bb5a8611a576568816"
server_key_version: "encryptionkey@0"
I20260925 15:18:46.327978 9477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:46.343358 9477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:46.347357 9477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:46.347625 9477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:46.348472 9477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:46.349748 9477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:46.349846 9477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:46.349992 9477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:46.350062 9477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:46.382076 9477 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:46479
I20260925 15:18:46.382117 9604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:46479 every 8 connection(s)
I20260925 15:18:46.383333 9477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:46.383795 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9477
I20260925 15:18:46.383980 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:18:46.384595 9097 external_mini_cluster.cc:1428] Setting key 167f94dae3e9d471b9012d388707d075
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:46.403616 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.386061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:50421 (local address 127.8.226.126:36971)
0925 15:18:46.386230 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:46.386241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:46.387544 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 15:18:46.389678 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:46.389683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:46.389783 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:46.389886 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:46.391533 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.392287 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.393797 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.394179 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.394308 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.397547 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:46.397551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:46.397555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:46.397584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:46.400753 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.401454 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.401459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.401463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.401584 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.401979 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.401983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.401987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.402244 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:46.402389 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:46.403074 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 15:18:46.403307 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:18:46.404551 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.386623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36971 (local address 127.8.226.67:50421)
0925 15:18:46.387398 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:46.387443 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:46.389407 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:46.390167 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:46.390176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:46.390747 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:46.391379 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:46.391405 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.392434 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.392439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:46.393662 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:46.393673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.394387 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.394395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:46.394607 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.395251 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:46.395276 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:46.397370 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:46.400877 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:46.400885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:46.400904 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:46.401316 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:46.401693 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:46.401699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:46.401702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:46.401865 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:46.402362 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:46.402369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:46.402715 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 15:18:46.402988 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 15:18:46.403449 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260925 15:18:46.405771 9605 heartbeater.cc:347] Connected to a master server at 127.8.226.126:36971
I20260925 15:18:46.406050 9605 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:46.406708 9605 heartbeater.cc:511] Master 127.8.226.126:36971 requested a full tablet report, sending...
I20260925 15:18:46.407984 9149 ts_manager.cc:194] Registered new tserver with Master: 8c4571b089c5438aba7f81f6f76dc169 (127.8.226.67:46479)
I20260925 15:18:46.408758 9149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:50421
I20260925 15:18:46.412603 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:18:46.452913 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.439065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.8.226.65:33981)
0925 15:18:46.439625 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:46.439631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:46.439658 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:18:46.439769 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:46.439775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:46.439970 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:46.440089 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:46.441345 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.442255 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.443311 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.443791 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.443876 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.447507 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:46.447517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:46.447529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:46.447559 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:46.450409 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.451039 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.451046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.451049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.451165 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.451556 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.451561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.451564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.451853 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:46.452129 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:46.452369 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:18:46.452600 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
W20260925 15:18:46.454061 9253 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47608
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:18:46.494656 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.481548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.8.226.65:33981)
0925 15:18:46.481785 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:46.481792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:46.482100 (+ 308us) server_negotiation.cc:407] Connection header received
0925 15:18:46.482237 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:46.482241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:46.482308 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:46.482386 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:46.483604 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.484250 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.485489 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.485937 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.486028 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.489655 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:46.489659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:46.489662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:46.489689 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:46.492241 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.492833 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.492841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.492844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.493015 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.493364 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.493370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.493373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.493609 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:46.493857 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:46.494144 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:18:46.494345 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:18:46.496043 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9210
I20260925 15:18:46.505576 9338 heartbeater.cc:503] Master 127.8.226.126:36971 was elected leader, sending a full tablet report...
I20260925 15:18:46.508219 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.497118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.8.226.65:33981)
0925 15:18:46.497310 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:46.497315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:46.497512 (+ 197us) server_negotiation.cc:407] Connection header received
0925 15:18:46.497685 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:46.497690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:46.497753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:46.497845 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:46.498977 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.499627 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.500810 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.501251 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.501371 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.503286 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:46.503290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:46.503293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:46.503316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:46.505876 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.506572 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.506576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.506579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.506715 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.507072 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.507076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.507080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.507339 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:46.507493 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:46.507806 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:18:46.507960 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:18:46.508858 9333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:46.587888 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9210
I20260925 15:18:46.602901 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9344
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:18:46.617877 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.603694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.8.226.66:42383)
0925 15:18:46.604283 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:46.604289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:46.604313 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:18:46.604370 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:46.604376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:46.604620 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:46.604813 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:46.606087 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.607099 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.608205 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.608712 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.608847 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.612628 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:46.612638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:46.612651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:46.612692 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:46.615167 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.615757 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.615764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.615766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.615887 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.616275 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.616281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.616287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.616595 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:46.616906 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:46.617268 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 15:18:46.617543 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":332,"threads_started":1}
I20260925 15:18:46.619035 9467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:46.697795 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9344
I20260925 15:18:46.712546 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9477
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:18:46.727160 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:46.713603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.8.226.67:46479)
0925 15:18:46.714224 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:46.714236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:46.714265 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:18:46.714326 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:46.714333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:46.714532 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:46.714652 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:46.715750 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.716757 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.717994 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:46.718519 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:46.718611 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:46.722294 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:46.722303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:46.722316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:46.722344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:46.724870 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.725489 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.725495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.725497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.725611 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:46.725904 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:46.725907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:46.725909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:46.726200 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:46.726418 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:46.726620 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:18:46.726817 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
I20260925 15:18:46.728339 9600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:46.805426 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9477
I20260925 15:18:46.819962 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9117
I20260925 15:18:46.821142 9179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:46.919772 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9117
2026-09-25T15:18:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3690 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9626](info): set up 2 sockets
Sep 25 15:18:46 dist-test-slave-8vw7 krb5kdc[9626](info): commencing operation
krb5kdc: starting...
W20260925 15:18:47.940733 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:18:48 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:18:48.100076 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39743
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:39743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:48.405071 9639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:48.405319 9639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:48.405463 9639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:48.412268 9639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:48.412356 9639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:48.412398 9639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:48.412470 9639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:48.412528 9639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:48.422016 9639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:39743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:48.423332 9639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:48.424475 9639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:48.436111 9647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:48.436149 9644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:48.436196 9645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:48.437243 9639 server_base.cc:1034] running on GCE node
I20260925 15:18:48.437886 9639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:48.439079 9639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:48.440294 9639 hybrid_clock.cc:648] HybridClock initialized: now 1790349528440254 us; error 42 us; skew 500 ppm
I20260925 15:18:48.440582 9639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:48 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:48.447602 9639 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:18:48.448904 9639 webserver.cc:460] Webserver started at http://127.8.226.126:39761/ using document root <none> and password file <none>
I20260925 15:18:48.449519 9639 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:48.449658 9639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:48.449939 9639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:48.452450 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "efab9f3bc51e4ebea2271a9a6cff6dce"
format_stamp: "Formatted at 2026-09-25 15:18:48 on dist-test-slave-8vw7"
server_key: "950d89bfe862e2e457c32205f1514673"
server_key_iv: "56523320efdbfe3a9c3fced076182003"
server_key_version: "encryptionkey@0"
I20260925 15:18:48.453158 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "efab9f3bc51e4ebea2271a9a6cff6dce"
format_stamp: "Formatted at 2026-09-25 15:18:48 on dist-test-slave-8vw7"
server_key: "950d89bfe862e2e457c32205f1514673"
server_key_iv: "56523320efdbfe3a9c3fced076182003"
server_key_version: "encryptionkey@0"
I20260925 15:18:48.457811 9639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:18:48.460835 9654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:48.462008 9639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:18:48.462204 9639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "efab9f3bc51e4ebea2271a9a6cff6dce"
format_stamp: "Formatted at 2026-09-25 15:18:48 on dist-test-slave-8vw7"
server_key: "950d89bfe862e2e457c32205f1514673"
server_key_iv: "56523320efdbfe3a9c3fced076182003"
server_key_version: "encryptionkey@0"
I20260925 15:18:48.462419 9639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:48.488348 9639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:48.491994 9639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:48.492264 9639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:48.509240 9639 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:39743
I20260925 15:18:48.509246 9705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:39743 every 8 connection(s)
I20260925 15:18:48.510519 9639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:48.513260 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9639
I20260925 15:18:48.513489 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:18:48.513995 9706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:48.514173 9097 external_mini_cluster.cc:1428] Setting key bf27a395c248c8ce7de9082fdb7b6c59
I20260925 15:18:48.526924 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce: Bootstrap starting.
Sep 25 15:18:48 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:48.530027 9706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:48.531049 9706 log.cc:824] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:48.533610 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce: No bootstrap required, opened a new log
I20260925 15:18:48.535324 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:48.516043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.8.226.126:39743)
0925 15:18:48.516872 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:48.516883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:48.516917 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:18:48.518371 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:48.518390 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:48.518728 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:48.519007 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:48.520590 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:48.521863 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:48.523613 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:48.524286 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:48.524414 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:48.528723 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:48.528730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:48.528743 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:48.528773 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:48.531685 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:48.532290 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:48.532304 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:48.532309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:48.532450 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:48.532803 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:48.532810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:48.532813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:48.533355 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:48.533633 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:48.534108 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 15:18:48.534489 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":382,"threads_started":1}
I20260925 15:18:48.541288 9706 raft_consensus.cc:348] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39743 } }
I20260925 15:18:48.541682 9706 raft_consensus.cc:374] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:48.541807 9706 raft_consensus.cc:729] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efab9f3bc51e4ebea2271a9a6cff6dce, State: Initialized, Role: FOLLOWER
I20260925 15:18:48.542441 9706 consensus_queue.cc:260] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39743 } }
I20260925 15:18:48.542642 9706 raft_consensus.cc:388] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:48.542728 9706 raft_consensus.cc:482] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:48.542843 9706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:48.544566 9706 raft_consensus.cc:504] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39743 } }
I20260925 15:18:48.544999 9706 leader_election.cc:302] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efab9f3bc51e4ebea2271a9a6cff6dce; no voters:
I20260925 15:18:48.545508 9706 leader_election.cc:288] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:48.545598 9711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:48.546262 9711 raft_consensus.cc:686] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [term 1 LEADER]: Becoming Leader. State: Replica: efab9f3bc51e4ebea2271a9a6cff6dce, State: Running, Role: LEADER
I20260925 15:18:48.546856 9711 consensus_queue.cc:237] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39743 } }
I20260925 15:18:48.547500 9706 sys_catalog.cc:564] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:48.549422 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader efab9f3bc51e4ebea2271a9a6cff6dce. Latest consensus state: current_term: 1 leader_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39743 } } }
I20260925 15:18:48.549793 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:48.550603 9718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:48.550472 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efab9f3bc51e4ebea2271a9a6cff6dce" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39743 } } }
I20260925 15:18:48.551613 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:48.555656 9718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:48.566803 9718 catalog_manager.cc:1347] Generated new cluster ID: f6b975b3df0641f8a95bca992aec1142
I20260925 15:18:48.566900 9718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:48.588761 9718 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:48.589962 9718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:48.602512 9718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efab9f3bc51e4ebea2271a9a6cff6dce: Generated new TSK 0
I20260925 15:18:48.603237 9718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:18:48.648598 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39743
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:48.960101 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:48.960466 9732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:48.960680 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:48.968343 9732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:48.968468 9732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:48.968631 9732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:18:48.977607 9732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39743
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:48.978744 9732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:48.979926 9732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:48.992246 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:48.992204 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:48.992291 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:48.993276 9732 server_base.cc:1034] running on GCE node
I20260925 15:18:48.993773 9732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:48.994561 9732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:48.995779 9732 hybrid_clock.cc:648] HybridClock initialized: now 1790349528995727 us; error 39 us; skew 500 ppm
I20260925 15:18:48.996068 9732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:48 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:48.998843 9732 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:18:49.000152 9732 webserver.cc:460] Webserver started at http://127.8.226.65:42957/ using document root <none> and password file <none>
I20260925 15:18:49.000813 9732 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:49.000942 9732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:49.001277 9732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:49.003890 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "411e76dc1c014edfa1d21d3c05c68064"
format_stamp: "Formatted at 2026-09-25 15:18:48 on dist-test-slave-8vw7"
server_key: "e7cb9cbbd6cc653ce85f298db39f5be2"
server_key_iv: "f60c35a796e7b7ccb6d4e1fe8f5b30f0"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.004575 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "411e76dc1c014edfa1d21d3c05c68064"
format_stamp: "Formatted at 2026-09-25 15:18:48 on dist-test-slave-8vw7"
server_key: "e7cb9cbbd6cc653ce85f298db39f5be2"
server_key_iv: "f60c35a796e7b7ccb6d4e1fe8f5b30f0"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.009137 9732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260925 15:18:49.012401 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.013671 9732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:49.013833 9732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "411e76dc1c014edfa1d21d3c05c68064"
format_stamp: "Formatted at 2026-09-25 15:18:48 on dist-test-slave-8vw7"
server_key: "e7cb9cbbd6cc653ce85f298db39f5be2"
server_key_iv: "f60c35a796e7b7ccb6d4e1fe8f5b30f0"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.014047 9732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:49.039189 9732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:49.044168 9732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:49.044447 9732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:49.045470 9732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:49.046635 9732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:49.046703 9732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.046789 9732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:49.046832 9732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.076973 9732 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:37459
I20260925 15:18:49.077029 9859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:37459 every 8 connection(s)
I20260925 15:18:49.078130 9732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:49.084321 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9732
I20260925 15:18:49.084517 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:18:49.085135 9097 external_mini_cluster.cc:1428] Setting key cde1b691fce64f16c27503a799b571c8
Sep 25 15:18:49 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:49.098735 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:49.080646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:46251 (local address 127.8.226.126:39743)
0925 15:18:49.081096 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:49.081100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:49.081992 (+ 892us) server_negotiation.cc:407] Connection header received
0925 15:18:49.083616 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:49.083621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:49.083702 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:49.083810 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:49.085299 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.086276 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.087997 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.088711 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.088851 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:49.092050 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:49.092076 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:49.092084 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:49.092136 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:49.095527 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:49.096225 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:49.096233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:49.096240 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:49.096383 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:49.096778 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:49.096786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:49.096791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:49.097217 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:49.097361 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:49.097989 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 15:18:49.098241 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260925 15:18:49.099429 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:49.081179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39743 (local address 127.8.226.65:46251)
0925 15:18:49.081834 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:49.081881 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:49.083374 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:49.084013 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:49.084021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:49.084534 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:49.085113 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:49.085130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.086444 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.086449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:49.087854 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:49.087868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.088900 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.088904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:49.089125 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:49.089766 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:49.089786 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:49.091837 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:49.095693 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:49.095699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:49.095710 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:49.096088 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:49.096499 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:49.096502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:49.096504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:49.096664 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:49.097343 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:49.097348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:49.097640 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 15:18:49.097841 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 15:18:49.098166 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":247,"threads_started":1}
I20260925 15:18:49.100865 9860 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39743
I20260925 15:18:49.101228 9860 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:49.102105 9860 heartbeater.cc:511] Master 127.8.226.126:39743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:18:49.104143 9671 ts_manager.cc:194] Registered new tserver with Master: 411e76dc1c014edfa1d21d3c05c68064 (127.8.226.65:37459)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
I20260925 15:18:49.105592 9671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:46251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:18:49.123896 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39743
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:49.422607 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:49.422837 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:49.422919 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:49.430897 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:49.430999 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:49.431138 9866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:18:49.440665 9866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39743
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:49.441811 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:49.442994 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:49.455039 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:49.455078 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:49.455112 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:49.455875 9866 server_base.cc:1034] running on GCE node
I20260925 15:18:49.456463 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:49.457353 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:49.458570 9866 hybrid_clock.cc:648] HybridClock initialized: now 1790349529458530 us; error 49 us; skew 500 ppm
I20260925 15:18:49.458858 9866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:49 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:49.461783 9866 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:18:49.463111 9866 webserver.cc:460] Webserver started at http://127.8.226.66:39639/ using document root <none> and password file <none>
I20260925 15:18:49.463770 9866 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:49.463901 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:49.464184 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:49.467089 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "b9585a2e25a748abbcc6c45bf2416588"
format_stamp: "Formatted at 2026-09-25 15:18:49 on dist-test-slave-8vw7"
server_key: "82c14f1f12084796f8081676ef336f51"
server_key_iv: "10f511ca14b82992ba7c77b402a3a377"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.467808 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "b9585a2e25a748abbcc6c45bf2416588"
format_stamp: "Formatted at 2026-09-25 15:18:49 on dist-test-slave-8vw7"
server_key: "82c14f1f12084796f8081676ef336f51"
server_key_iv: "10f511ca14b82992ba7c77b402a3a377"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.472263 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:18:49.475373 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.476611 9866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:49.476778 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "b9585a2e25a748abbcc6c45bf2416588"
format_stamp: "Formatted at 2026-09-25 15:18:49 on dist-test-slave-8vw7"
server_key: "82c14f1f12084796f8081676ef336f51"
server_key_iv: "10f511ca14b82992ba7c77b402a3a377"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.476986 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:49.496126 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:49.499568 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:49.499838 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:49.500752 9866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:49.502002 9866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:49.502103 9866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.502251 9866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:49.502324 9866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.536077 9866 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:36253
I20260925 15:18:49.536160 9993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:36253 every 8 connection(s)
I20260925 15:18:49.537382 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:49.547777 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9866
I20260925 15:18:49.547961 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:18:49.548357 9097 external_mini_cluster.cc:1428] Setting key a8eb653538226dbcd2223c5cc519457b
Sep 25 15:18:49 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:49.555768 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:49.539624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:42241 (local address 127.8.226.126:39743)
0925 15:18:49.539816 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:49.539822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:49.540899 (+ 1077us) server_negotiation.cc:407] Connection header received
0925 15:18:49.542504 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:49.542508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:49.542566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:49.542648 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:49.544021 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.544678 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.546186 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.546571 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.546668 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:49.550305 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:49.550312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:49.550316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:49.550349 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:49.552716 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:49.553425 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:49.553430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:49.553434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:49.553574 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:49.554011 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:49.554017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:49.554020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:49.554357 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:49.554519 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:49.555184 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 15:18:49.555412 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:18:49.556708 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:49.540092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39743 (local address 127.8.226.66:42241)
0925 15:18:49.540742 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:49.540783 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:49.542285 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:49.542825 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:49.542833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:49.543326 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:49.543873 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:49.543888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.544831 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.544834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:49.546045 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:49.546057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:49.546753 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:49.546758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:49.546948 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:49.547601 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:49.547638 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:49.550091 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:49.552848 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:49.552856 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:49.552869 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:49.553309 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:49.553691 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:49.553696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:49.553700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:49.553900 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:49.554488 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:49.554494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:49.554835 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 15:18:49.555044 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 15:18:49.555347 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":244,"threads_started":1}
I20260925 15:18:49.557963 9994 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39743
I20260925 15:18:49.558257 9994 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:49.558885 9994 heartbeater.cc:511] Master 127.8.226.126:39743 requested a full tablet report, sending...
I20260925 15:18:49.560228 9671 ts_manager.cc:194] Registered new tserver with Master: b9585a2e25a748abbcc6c45bf2416588 (127.8.226.66:36253)
I20260925 15:18:49.561038 9671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:42241
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:18:49.585767 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39743
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:49.881145 9999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:49.881392 9999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:49.881477 9999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:49.888015 9999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:49.888110 9999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:49.888236 9999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:18:49.897475 9999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39743
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:49.898567 9999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:49.899783 9999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:49.912082 10004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:49.912127 10005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:49.912173 10007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:49.912892 9999 server_base.cc:1034] running on GCE node
I20260925 15:18:49.913496 9999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:49.914353 9999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:49.915570 9999 hybrid_clock.cc:648] HybridClock initialized: now 1790349529915529 us; error 54 us; skew 500 ppm
I20260925 15:18:49.915863 9999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:49 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:49.919024 9999 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:18:49.920259 9999 webserver.cc:460] Webserver started at http://127.8.226.67:43719/ using document root <none> and password file <none>
I20260925 15:18:49.920871 9999 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:49.921026 9999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:49.921327 9999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:49.923925 9999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "6cd7ca7290f74e9b8a6b2fe494eae543"
format_stamp: "Formatted at 2026-09-25 15:18:49 on dist-test-slave-8vw7"
server_key: "7882fba1de67348f575ccddfab9c16f1"
server_key_iv: "81bb5b10db3e7a9a8880661b378bf052"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.924661 9999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "6cd7ca7290f74e9b8a6b2fe494eae543"
format_stamp: "Formatted at 2026-09-25 15:18:49 on dist-test-slave-8vw7"
server_key: "7882fba1de67348f575ccddfab9c16f1"
server_key_iv: "81bb5b10db3e7a9a8880661b378bf052"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.929467 9999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 15:18:49.932475 10014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.933698 9999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:18:49.933868 9999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "6cd7ca7290f74e9b8a6b2fe494eae543"
format_stamp: "Formatted at 2026-09-25 15:18:49 on dist-test-slave-8vw7"
server_key: "7882fba1de67348f575ccddfab9c16f1"
server_key_iv: "81bb5b10db3e7a9a8880661b378bf052"
server_key_version: "encryptionkey@0"
I20260925 15:18:49.934067 9999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:49.963017 9999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:49.967049 9999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:49.967271 9999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:49.968075 9999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:49.969318 9999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:49.969386 9999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.969465 9999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:49.969511 9999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:49.999195 9999 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:44667
I20260925 15:18:49.999289 10126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:44667 every 8 connection(s)
I20260925 15:18:50.000398 9999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:50.009013 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 9999
I20260925 15:18:50.009198 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:18:50.009718 9097 external_mini_cluster.cc:1428] Setting key 52a8d18bf44d1ea57d76e7f581b63cdb
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:50.019711 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.003326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:45441 (local address 127.8.226.126:39743)
0925 15:18:50.003544 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.003548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.004603 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 15:18:50.006337 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.006341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.006405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.006490 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.007849 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.008557 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.010753 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.011286 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.011374 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.014881 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.014886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.014889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.014919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.017251 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.017882 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.017885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.017887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.018017 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.018432 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.018435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.018437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.018670 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.018769 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.019330 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.019483 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 15:18:50.020596 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.003812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39743 (local address 127.8.226.67:45441)
0925 15:18:50.004434 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:50.004468 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:50.006089 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:50.006686 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:50.006705 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:50.007179 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:50.007705 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.007721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.008724 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.008730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.010582 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.010595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.011509 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.011514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.011823 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.012548 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:50.012574 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:50.014709 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:50.017387 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:50.017393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:50.017405 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:50.017791 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:50.018151 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:50.018158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:50.018162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:50.018337 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:50.018765 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:50.018770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:50.019037 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 15:18:50.019238 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 15:18:50.019572 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":245,"threads_started":1}
I20260925 15:18:50.021929 10127 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39743
I20260925 15:18:50.022203 10127 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:50.022853 10127 heartbeater.cc:511] Master 127.8.226.126:39743 requested a full tablet report, sending...
I20260925 15:18:50.024441 9671 ts_manager.cc:194] Registered new tserver with Master: 6cd7ca7290f74e9b8a6b2fe494eae543 (127.8.226.67:44667)
I20260925 15:18:50.025085 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:50.025416 9671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:45441
I20260925 15:18:50.043504 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.031502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.8.226.126:39743)
0925 15:18:50.031691 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.031699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.032135 (+ 436us) server_negotiation.cc:407] Connection header received
0925 15:18:50.032267 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.032271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.032338 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.032429 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.034057 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.034765 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.036066 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.036477 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.036569 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.038978 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.038983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.038986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.039013 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.041365 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.041937 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.041941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.041943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.042054 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.042333 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.042336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.042338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.042603 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.042772 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.043030 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.043167 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:18:50.049857 9671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:50.056334 9671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:50.089016 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.067824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.8.226.67:44667)
0925 15:18:50.068595 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.068607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.081048 (+ 12441us) server_negotiation.cc:407] Connection header received
0925 15:18:50.081371 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.081380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.081651 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.081803 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:50.082725 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.084209 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.086336 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.087689 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.087846 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.088053 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.088536 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.088687 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":548,"threads_started":1}
I20260925 15:18:50.089041 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.067706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:44667 (local address 127.0.0.1:36126)
0925 15:18:50.080740 (+ 13034us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:50.080781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:50.081121 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:50.082103 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:50.082110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:50.082141 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:50.082528 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.082541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.084344 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.084351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.086151 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.086165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.088093 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.088100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.088315 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.088377 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 15:18:50.088497 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:18:50.088660 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12827,"thread_start_us":14913,"threads_started":1}
I20260925 15:18:50.091929 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.070434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36253 (local address 127.0.0.1:53480)
0925 15:18:50.083726 (+ 13292us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:50.083742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:50.083850 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:50.084864 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:50.084871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:50.084902 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:50.085430 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.085443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.087257 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.087263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.089210 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.089224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.091177 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.091183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.091365 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.091406 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:18:50.091489 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:18:50.091605 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13154,"thread_start_us":13304,"threads_started":1}
I20260925 15:18:50.092036 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.071063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.8.226.66:36253)
0925 15:18:50.083342 (+ 12279us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.083350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.083965 (+ 615us) server_negotiation.cc:407] Connection header received
0925 15:18:50.084081 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.084090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.084346 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.084575 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:50.085636 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.087105 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.089382 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.090930 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.091068 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.091264 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.091567 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.091713 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12100,"thread_start_us":12158,"threads_started":1}
I20260925 15:18:50.093144 10062 tablet_service.cc:1467] Processing CreateTablet for tablet 114b99acc0f74a7384ba26f2a9c68c57 (DEFAULT_TABLE table=test-table [id=b155083f71a74bb5afe579a05a368333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:50.094411 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.067760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37459 (local address 127.0.0.1:52892)
0925 15:18:50.082366 (+ 14606us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:50.082389 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:50.082524 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:50.087706 (+ 5182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:50.087710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:50.087742 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:50.088206 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.088216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.089921 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.089926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.091890 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.091903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.093725 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.093743 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.093911 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.093940 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:18:50.094003 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:18:50.094123 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14454,"thread_start_us":12184,"threads_started":1}
I20260925 15:18:50.094753 10062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b99acc0f74a7384ba26f2a9c68c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:50.095360 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.068212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.8.226.65:37459)
0925 15:18:50.087321 (+ 19109us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.087326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.087346 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:18:50.087405 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.087409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.087540 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.087719 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:50.088369 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.089754 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.092068 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.093503 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.094818 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.095008 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.095055 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.095155 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18968,"thread_start_us":18840,"threads_started":1}
I20260925 15:18:50.095500 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 114b99acc0f74a7384ba26f2a9c68c57 (DEFAULT_TABLE table=test-table [id=b155083f71a74bb5afe579a05a368333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:50.096839 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b99acc0f74a7384ba26f2a9c68c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:50.100865 9795 tablet_service.cc:1467] Processing CreateTablet for tablet 114b99acc0f74a7384ba26f2a9c68c57 (DEFAULT_TABLE table=test-table [id=b155083f71a74bb5afe579a05a368333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:50.101970 9795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b99acc0f74a7384ba26f2a9c68c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:50.108155 9860 heartbeater.cc:503] Master 127.8.226.126:39743 was elected leader, sending a full tablet report...
I20260925 15:18:50.109066 10145 tablet_bootstrap.cc:492] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: Bootstrap starting.
I20260925 15:18:50.112725 10145 tablet_bootstrap.cc:654] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:50.113106 10146 tablet_bootstrap.cc:492] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: Bootstrap starting.
I20260925 15:18:50.113889 10147 tablet_bootstrap.cc:492] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: Bootstrap starting.
I20260925 15:18:50.114159 10145 log.cc:824] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:50.116636 10146 tablet_bootstrap.cc:654] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:50.117372 10147 tablet_bootstrap.cc:654] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:50.117873 10146 log.cc:824] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:50.118494 10147 log.cc:824] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:50.119438 10145 tablet_bootstrap.cc:492] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: No bootstrap required, opened a new log
I20260925 15:18:50.119715 10145 ts_tablet_manager.cc:1397] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260925 15:18:50.120936 10147 tablet_bootstrap.cc:492] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: No bootstrap required, opened a new log
I20260925 15:18:50.121189 10147 ts_tablet_manager.cc:1397] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:18:50.121508 10146 tablet_bootstrap.cc:492] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: No bootstrap required, opened a new log
I20260925 15:18:50.121730 10146 ts_tablet_manager.cc:1397] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 15:18:50.126497 10145 raft_consensus.cc:348] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.126890 10145 raft_consensus.cc:374] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:50.127017 10145 raft_consensus.cc:729] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd7ca7290f74e9b8a6b2fe494eae543, State: Initialized, Role: FOLLOWER
I20260925 15:18:50.127604 10145 consensus_queue.cc:260] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.128657 10127 heartbeater.cc:503] Master 127.8.226.126:39743 was elected leader, sending a full tablet report...
I20260925 15:18:50.129235 10145 ts_tablet_manager.cc:1428] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 15:18:50.130847 10147 raft_consensus.cc:348] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.131191 10147 raft_consensus.cc:374] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:50.131304 10147 raft_consensus.cc:729] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411e76dc1c014edfa1d21d3c05c68064, State: Initialized, Role: FOLLOWER
I20260925 15:18:50.131841 10147 consensus_queue.cc:260] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.132067 10146 raft_consensus.cc:348] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.132513 10146 raft_consensus.cc:374] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:50.132612 10146 raft_consensus.cc:729] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9585a2e25a748abbcc6c45bf2416588, State: Initialized, Role: FOLLOWER
I20260925 15:18:50.133144 10146 consensus_queue.cc:260] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.133463 10147 ts_tablet_manager.cc:1428] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 15:18:50.134246 9994 heartbeater.cc:503] Master 127.8.226.126:39743 was elected leader, sending a full tablet report...
I20260925 15:18:50.134769 10146 ts_tablet_manager.cc:1428] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:18:50.249372 10152 raft_consensus.cc:482] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:50.249641 10152 raft_consensus.cc:504] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.251204 10152 leader_election.cc:288] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cd7ca7290f74e9b8a6b2fe494eae543 (127.8.226.67:44667), b9585a2e25a748abbcc6c45bf2416588 (127.8.226.66:36253)
W20260925 15:18:50.252280 10128 tablet.cc:2369] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:50.257690 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.251586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:47667 (local address 127.8.226.66:36253)
0925 15:18:50.251782 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.251788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.252497 (+ 709us) server_negotiation.cc:407] Connection header received
0925 15:18:50.252860 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.252867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.252968 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.253052 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:50.253638 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.254343 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.256036 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.256787 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.256967 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.257058 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.257311 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.257442 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:18:50.257876 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.251606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:44717 (local address 127.8.226.67:44667)
0925 15:18:50.251798 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.251805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.252978 (+ 1173us) server_negotiation.cc:407] Connection header received
0925 15:18:50.253124 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.253128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.253207 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.253324 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:50.253889 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.254621 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.256265 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.257031 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.257186 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.257254 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.257531 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.257645 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:18:50.258065 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.251439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:44667 (local address 127.8.226.65:44717)
0925 15:18:50.252844 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:50.252860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:50.252994 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:50.253343 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:50.253347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:50.253373 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:50.253745 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.253753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.254741 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.254747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.256116 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.256127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.257219 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.257225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.257361 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.257399 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:18:50.257508 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:18:50.257624 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":111,"thread_start_us":1012,"threads_started":1}
I20260925 15:18:50.258039 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.251439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36253 (local address 127.8.226.65:47667)
0925 15:18:50.252386 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:50.252406 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:50.252617 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:50.253126 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:50.253130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:50.253151 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:50.253500 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.253508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.254468 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.254473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.255898 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:50.255909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.257011 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.257017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:50.257162 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.257182 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 15:18:50.257288 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:18:50.257421 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":50,"thread_start_us":556,"threads_started":1}
I20260925 15:18:50.258960 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b99acc0f74a7384ba26f2a9c68c57" candidate_uuid: "411e76dc1c014edfa1d21d3c05c68064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" is_pre_election: true
I20260925 15:18:50.259035 9949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b99acc0f74a7384ba26f2a9c68c57" candidate_uuid: "411e76dc1c014edfa1d21d3c05c68064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9585a2e25a748abbcc6c45bf2416588" is_pre_election: true
I20260925 15:18:50.259441 10082 raft_consensus.cc:2454] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411e76dc1c014edfa1d21d3c05c68064 in term 0.
I20260925 15:18:50.259474 9949 raft_consensus.cc:2454] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411e76dc1c014edfa1d21d3c05c68064 in term 0.
I20260925 15:18:50.260056 9748 leader_election.cc:302] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411e76dc1c014edfa1d21d3c05c68064, b9585a2e25a748abbcc6c45bf2416588; no voters:
I20260925 15:18:50.260399 10152 raft_consensus.cc:2781] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:50.260545 10152 raft_consensus.cc:482] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:50.260663 10152 raft_consensus.cc:3037] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:50.262725 10152 raft_consensus.cc:504] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.263245 10152 leader_election.cc:288] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [CANDIDATE]: Term 1 election: Requested vote from peers 6cd7ca7290f74e9b8a6b2fe494eae543 (127.8.226.67:44667), b9585a2e25a748abbcc6c45bf2416588 (127.8.226.66:36253)
I20260925 15:18:50.263583 10082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b99acc0f74a7384ba26f2a9c68c57" candidate_uuid: "411e76dc1c014edfa1d21d3c05c68064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543"
I20260925 15:18:50.263770 10082 raft_consensus.cc:3037] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:50.263743 9949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b99acc0f74a7384ba26f2a9c68c57" candidate_uuid: "411e76dc1c014edfa1d21d3c05c68064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9585a2e25a748abbcc6c45bf2416588"
I20260925 15:18:50.263931 9949 raft_consensus.cc:3037] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:50.266288 10082 raft_consensus.cc:2454] T 114b99acc0f74a7384ba26f2a9c68c57 P 6cd7ca7290f74e9b8a6b2fe494eae543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411e76dc1c014edfa1d21d3c05c68064 in term 1.
I20260925 15:18:50.266296 9949 raft_consensus.cc:2454] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411e76dc1c014edfa1d21d3c05c68064 in term 1.
I20260925 15:18:50.266670 9748 leader_election.cc:302] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411e76dc1c014edfa1d21d3c05c68064, b9585a2e25a748abbcc6c45bf2416588; no voters:
I20260925 15:18:50.266959 10152 raft_consensus.cc:2781] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:50.267379 10152 raft_consensus.cc:686] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [term 1 LEADER]: Becoming Leader. State: Replica: 411e76dc1c014edfa1d21d3c05c68064, State: Running, Role: LEADER
I20260925 15:18:50.267848 10152 consensus_queue.cc:237] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } }
I20260925 15:18:50.272212 9670 catalog_manager.cc:5495] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 411e76dc1c014edfa1d21d3c05c68064 (127.8.226.65). New cstate: current_term: 1 leader_uuid: "411e76dc1c014edfa1d21d3c05c68064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd7ca7290f74e9b8a6b2fe494eae543" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "411e76dc1c014edfa1d21d3c05c68064" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9585a2e25a748abbcc6c45bf2416588" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36253 } health_report { overall_health: UNKNOWN } } }
W20260925 15:18:50.289163 9995 tablet.cc:2369] T 114b99acc0f74a7384ba26f2a9c68c57 P b9585a2e25a748abbcc6c45bf2416588: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
W20260925 15:18:50.330346 9861 tablet.cc:2369] T 114b99acc0f74a7384ba26f2a9c68c57 P 411e76dc1c014edfa1d21d3c05c68064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:50.331131 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.317012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.8.226.65:37459)
0925 15:18:50.317169 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.317193 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.317657 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:18:50.317852 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.317856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.317914 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.318007 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.319488 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.320208 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.321451 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.321868 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.321989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.325627 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.325641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.325655 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.325684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.328439 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.329082 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.329089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.329091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.329240 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.329606 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.329610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.329612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.329951 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.330255 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.330553 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.330835 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349530, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349530, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:18:50.372263 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.359831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.8.226.65:37459)
0925 15:18:50.360056 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.360063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.360361 (+ 298us) server_negotiation.cc:407] Connection header received
0925 15:18:50.360592 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.360596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.360652 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.360760 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.362030 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.362722 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.363984 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.364354 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.364493 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.367966 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.367970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.367974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.368002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.370479 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.370968 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.370971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.370973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.371082 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.371350 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.371352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.371354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.371560 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.371757 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.371907 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.372047 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:18:50.412653 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.399906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.8.226.65:37459)
0925 15:18:50.400106 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.400114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.400276 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:18:50.400413 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.400417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.400499 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.400577 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.401896 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.402663 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.403822 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.404223 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.404319 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.407967 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.407971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.407974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.408005 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.410642 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.411163 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.411166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.411169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.411309 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.411571 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.411574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.411575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.411807 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.411985 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.412132 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.412277 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:18:50.414361 9775 tablet_service.cc:2908] Rejecting scan request for tablet 114b99acc0f74a7384ba26f2a9c68c57: Uninitialized: clean time has not yet been initialized
I20260925 15:18:50.417209 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9732
I20260925 15:18:50.429104 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.417986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.8.226.65:37459)
0925 15:18:50.418125 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.418146 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.418532 (+ 386us) server_negotiation.cc:407] Connection header received
0925 15:18:50.418646 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.418649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.418696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.418772 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.419767 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.420412 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.421545 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.421892 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.421975 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.423765 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.423769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.423772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.423797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.426033 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.426783 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.426788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.426791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.426946 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.427402 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.427407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.427411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.427729 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.427976 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.428330 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.428575 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:18:50.429812 9855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:50.516547 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9732
I20260925 15:18:50.533308 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9866
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:18:50.549185 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.533998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.8.226.66:36253)
0925 15:18:50.534153 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.534157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.534295 (+ 138us) server_negotiation.cc:407] Connection header received
0925 15:18:50.534445 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.534450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.534519 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.534625 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.535925 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.536727 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.538516 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.538990 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.539085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.543527 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.543536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.543549 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.543575 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.546394 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.547135 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.547141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.547143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.547257 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.547585 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.547588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.547589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.548083 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.548316 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.548562 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.548749 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:18:50.549877 9989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:50.631300 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9866
I20260925 15:18:50.646452 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9999
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[9626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:18:50.661068 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:50.647213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.8.226.67:44667)
0925 15:18:50.647395 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:50.647400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:50.647421 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:18:50.647617 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:50.647621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:50.647683 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:50.647796 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:50.649578 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.650339 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.651561 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:50.651944 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:50.652025 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:50.655938 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:50.655950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:50.655965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:50.655999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:50.658416 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.659048 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.659056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.659058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.659184 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:50.659524 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:50.659530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:50.659533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:50.659868 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:50.660153 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:50.660466 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:18:50.660664 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:18:50.661702 10122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:50.746042 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9999
I20260925 15:18:50.760758 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 9639
I20260925 15:18:50.761956 9701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:50.870347 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 9639
2026-09-25T15:18:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3954 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[10175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[10175](info): set up 2 sockets
Sep 25 15:18:50 dist-test-slave-8vw7 krb5kdc[10175](info): commencing operation
krb5kdc: starting...
W20260925 15:18:51.888736 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:18:51 dist-test-slave-8vw7 krb5kdc[10175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:18:52.055974 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34749
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:52.356678 10188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:52.356980 10188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:52.357067 10188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:52.363727 10188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:52.363818 10188 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:18:52.363857 10188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:52.363931 10188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:52.363976 10188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:52.364030 10188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:52.373186 10188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:34749
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:52.374359 10188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:52.375666 10188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:52.387389 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:52.387498 10196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:52.387442 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:52.388540 10188 server_base.cc:1034] running on GCE node
I20260925 15:18:52.389384 10188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:52.390678 10188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:52.391911 10188 hybrid_clock.cc:648] HybridClock initialized: now 1790349532391859 us; error 32 us; skew 500 ppm
I20260925 15:18:52.392236 10188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:52 dist-test-slave-8vw7 krb5kdc[10175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349532, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:52.395376 10188 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:18:52.396790 10188 webserver.cc:460] Webserver started at http://127.8.226.126:41097/ using document root <none> and password file <none>
I20260925 15:18:52.397507 10188 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:52.397642 10188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:52.397989 10188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:52.400820 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "d5644564aecc49e58b5dcafd171102cf"
format_stamp: "Formatted at 2026-09-25 15:18:52 on dist-test-slave-8vw7"
server_key: "dfdcb07fb7faeee00c6ee9a6e2bdb3c4"
server_key_iv: "0e7822d9a719d2352193fe9755d01949"
server_key_version: "encryptionkey@0"
I20260925 15:18:52.401598 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "d5644564aecc49e58b5dcafd171102cf"
format_stamp: "Formatted at 2026-09-25 15:18:52 on dist-test-slave-8vw7"
server_key: "dfdcb07fb7faeee00c6ee9a6e2bdb3c4"
server_key_iv: "0e7822d9a719d2352193fe9755d01949"
server_key_version: "encryptionkey@0"
I20260925 15:18:52.406194 10188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:18:52.409479 10203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:52.410774 10188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:18:52.410950 10188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "d5644564aecc49e58b5dcafd171102cf"
format_stamp: "Formatted at 2026-09-25 15:18:52 on dist-test-slave-8vw7"
server_key: "dfdcb07fb7faeee00c6ee9a6e2bdb3c4"
server_key_iv: "0e7822d9a719d2352193fe9755d01949"
server_key_version: "encryptionkey@0"
I20260925 15:18:52.411144 10188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:52.422505 10188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:52.426433 10188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:52.426689 10188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:52.447285 10188 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:34749
I20260925 15:18:52.447305 10264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:34749 every 8 connection(s)
I20260925 15:18:52.448573 10188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:52.451216 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 10188
I20260925 15:18:52.451455 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:18:52.452286 9097 external_mini_cluster.cc:1428] Setting key f5f69a559dd0c4ca2644c38cc89799ee
I20260925 15:18:52.452808 10265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:52.464380 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: Bootstrap starting.
Sep 25 15:18:52 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:52.468164 10265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:52.469228 10265 log.cc:824] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:52.472852 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: No bootstrap required, opened a new log
I20260925 15:18:52.479202 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:52.454332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.8.226.126:34749)
0925 15:18:52.455042 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:52.455054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:52.455091 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:18:52.457347 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:52.457366 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:52.457744 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:52.458124 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:52.459570 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:52.460838 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:52.462042 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:52.462750 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:52.462937 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:52.467470 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:52.467480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:52.467495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:52.467537 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:52.474972 (+ 7435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:52.475598 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:52.475608 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:52.475622 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:52.475835 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:52.476178 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:52.476184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:52.476193 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:52.476837 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:52.477169 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:52.477522 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 15:18:52.477921 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":335,"threads_started":1}
I20260925 15:18:52.482623 10265 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5644564aecc49e58b5dcafd171102cf" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34749 } }
I20260925 15:18:52.482916 10265 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:52.483017 10265 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5644564aecc49e58b5dcafd171102cf, State: Initialized, Role: FOLLOWER
I20260925 15:18:52.483516 10265 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5644564aecc49e58b5dcafd171102cf" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34749 } }
I20260925 15:18:52.483709 10265 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:52.483801 10265 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:52.483915 10265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:52.486325 10265 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5644564aecc49e58b5dcafd171102cf" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34749 } }
I20260925 15:18:52.486734 10265 leader_election.cc:302] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5644564aecc49e58b5dcafd171102cf; no voters:
I20260925 15:18:52.487212 10265 leader_election.cc:288] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:52.487372 10270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:52.488050 10270 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [term 1 LEADER]: Becoming Leader. State: Replica: d5644564aecc49e58b5dcafd171102cf, State: Running, Role: LEADER
I20260925 15:18:52.488615 10270 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5644564aecc49e58b5dcafd171102cf" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34749 } }
I20260925 15:18:52.489336 10265 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:52.491724 10271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5644564aecc49e58b5dcafd171102cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5644564aecc49e58b5dcafd171102cf" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34749 } } }
I20260925 15:18:52.492127 10271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:52.492830 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5644564aecc49e58b5dcafd171102cf. Latest consensus state: current_term: 1 leader_uuid: "d5644564aecc49e58b5dcafd171102cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5644564aecc49e58b5dcafd171102cf" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34749 } } }
I20260925 15:18:52.493084 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:52.493191 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:52.507356 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:52.516194 10279 catalog_manager.cc:1347] Generated new cluster ID: c3eff072a7c342bb89095362a34dfd53
I20260925 15:18:52.516311 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:52.532601 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:52.534379 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:52.547279 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: Generated new TSK 0
I20260925 15:18:52.548220 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:18:52.596091 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34749
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:52.895818 10292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:52.896100 10292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:52.896265 10292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:52.903352 10292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:52.903468 10292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:52.903613 10292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:18:52.903743 10292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:18:52.913304 10292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:34749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:52.914542 10292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:52.915881 10292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:52.928185 10298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:52.928248 10297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:52.928267 10300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:52.928747 10292 server_base.cc:1034] running on GCE node
I20260925 15:18:52.929351 10292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:52.930238 10292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:52.931488 10292 hybrid_clock.cc:648] HybridClock initialized: now 1790349532931433 us; error 39 us; skew 500 ppm
I20260925 15:18:52.931792 10292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:52 dist-test-slave-8vw7 krb5kdc[10175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349532, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:52.934773 10292 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:18:52.936129 10292 webserver.cc:460] Webserver started at http://127.8.226.65:35603/ using document root <none> and password file <none>
I20260925 15:18:52.936779 10292 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:52.936902 10292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:52.937225 10292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:52.939980 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "c3448e6707a74a34a0c2955310e8b6be"
format_stamp: "Formatted at 2026-09-25 15:18:52 on dist-test-slave-8vw7"
server_key: "b93d0b03321a628126d4a1f4547cb7c6"
server_key_iv: "1c82f3eef3a5eac2400bd59a4bd7e068"
server_key_version: "encryptionkey@0"
I20260925 15:18:52.940821 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "c3448e6707a74a34a0c2955310e8b6be"
format_stamp: "Formatted at 2026-09-25 15:18:52 on dist-test-slave-8vw7"
server_key: "b93d0b03321a628126d4a1f4547cb7c6"
server_key_iv: "1c82f3eef3a5eac2400bd59a4bd7e068"
server_key_version: "encryptionkey@0"
I20260925 15:18:52.945449 10292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:18:52.948633 10307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:52.949903 10292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:52.950062 10292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "c3448e6707a74a34a0c2955310e8b6be"
format_stamp: "Formatted at 2026-09-25 15:18:52 on dist-test-slave-8vw7"
server_key: "b93d0b03321a628126d4a1f4547cb7c6"
server_key_iv: "1c82f3eef3a5eac2400bd59a4bd7e068"
server_key_version: "encryptionkey@0"
I20260925 15:18:52.950286 10292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:52.960935 10292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:52.964437 10292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:52.964684 10292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:52.967043 10292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:52.967152 10292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:52.967331 10292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:52.967417 10292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:18:52 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349532, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:52.989648 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:52.967789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:51969 (local address 127.8.226.126:34749)
0925 15:18:52.967969 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:52.967975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:52.969253 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 15:18:52.971452 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:52.971462 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:52.971570 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:52.971694 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:52.973778 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:52.975086 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:52.977504 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:52.978100 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:52.978234 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:52.982731 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:52.982736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:52.982739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:52.982773 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:52.986129 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:52.986839 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:52.986848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:52.986854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:52.987042 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:52.987529 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:52.987536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:52.987542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:52.988013 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:52.988199 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:52.988817 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 15:18:52.989142 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:18:52.990430 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:52.968235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.8.226.65:51969)
0925 15:18:52.969084 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:52.969131 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:52.971027 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:52.971959 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:52.971966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:52.972706 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:52.973544 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:52.973573 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:52.975286 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:52.975294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:52.977353 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:52.977364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:52.978324 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:52.978332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:52.978538 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:52.979269 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:52.979291 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:52.982487 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:52.986281 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:52.986292 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:52.986312 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:52.986707 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:52.987185 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:52.987188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:52.987190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:52.987411 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:52.988133 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:52.988138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:52.988458 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 15:18:52.988660 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 15:18:52.989071 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":373,"threads_started":1}
I20260925 15:18:53.012189 10292 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:42419
I20260925 15:18:53.012243 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:52.996880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:53403 (local address 127.8.226.126:34749)
0925 15:18:52.997120 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:52.997126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:52.998218 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 15:18:52.998512 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:52.998517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:52.998590 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:52.998690 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:52.999873 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.000902 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.002909 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.003277 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.003409 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.005730 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:53.005737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:53.005743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:53.005780 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:53.008901 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.009968 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.009973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.009976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.010153 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.010499 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.010507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.010513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.010926 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:53.011089 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:53.011583 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 15:18:53.011833 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 15:18:53.012502 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:52.997072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.8.226.65:53403)
0925 15:18:52.998071 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:52.998101 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:52.998342 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:52.998712 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:52.998716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:52.999357 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:52.999741 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:52.999750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.001078 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.001083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.002764 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.002775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.003477 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.003487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.003647 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.004269 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:53.004288 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:53.005476 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:53.009190 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.009196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.009199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.009831 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.010270 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.010274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.010276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.010387 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.011055 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:53.011060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:53.011265 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 15:18:53.011456 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:18:53.011851 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":704,"threads_started":1}
I20260925 15:18:53.012264 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:42419 every 8 connection(s)
I20260925 15:18:53.013839 10292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:53.016026 10427 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34749
I20260925 15:18:53.016288 10427 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:53.016892 10427 heartbeater.cc:511] Master 127.8.226.126:34749 requested a full tablet report, sending...
I20260925 15:18:53.018662 10220 ts_manager.cc:194] Registered new tserver with Master: c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419)
I20260925 15:18:53.020061 10220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:51969
I20260925 15:18:53.021620 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 10292
I20260925 15:18:53.021839 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:18:53.022436 9097 external_mini_cluster.cc:1428] Setting key 93172129183048ab0cfe8bde7e569dec
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:18:53.064072 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34749
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:53.366917 10431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:53.367189 10431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:53.367280 10431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:53.374089 10431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:53.374185 10431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:53.374262 10431 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:18:53.374339 10431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:18:53.383347 10431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:34749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:53.384569 10431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:53.385823 10431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:53.397756 10436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:53.397890 10437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:53.398202 10439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:53.402146 10431 server_base.cc:1034] running on GCE node
I20260925 15:18:53.402623 10431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:53.403360 10431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:53.404556 10431 hybrid_clock.cc:648] HybridClock initialized: now 1790349533404505 us; error 33 us; skew 500 ppm
I20260925 15:18:53.404827 10431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:53 dist-test-slave-8vw7 krb5kdc[10175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349533, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:53.408092 10431 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:18:53.409634 10431 webserver.cc:460] Webserver started at http://127.8.226.66:46855/ using document root <none> and password file <none>
I20260925 15:18:53.410296 10431 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:53.410403 10431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:53.410709 10431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:53.413412 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "e8bb45b03d494bdd951daaae74cbb784"
format_stamp: "Formatted at 2026-09-25 15:18:53 on dist-test-slave-8vw7"
server_key: "329b35fba005887b43f25abbbfd7d3ea"
server_key_iv: "3926147f24772ad16a3ec305fa727625"
server_key_version: "encryptionkey@0"
I20260925 15:18:53.414206 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "e8bb45b03d494bdd951daaae74cbb784"
format_stamp: "Formatted at 2026-09-25 15:18:53 on dist-test-slave-8vw7"
server_key: "329b35fba005887b43f25abbbfd7d3ea"
server_key_iv: "3926147f24772ad16a3ec305fa727625"
server_key_version: "encryptionkey@0"
I20260925 15:18:53.418866 10431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:18:53.422096 10446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:53.423281 10431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:18:53.423432 10431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "e8bb45b03d494bdd951daaae74cbb784"
format_stamp: "Formatted at 2026-09-25 15:18:53 on dist-test-slave-8vw7"
server_key: "329b35fba005887b43f25abbbfd7d3ea"
server_key_iv: "3926147f24772ad16a3ec305fa727625"
server_key_version: "encryptionkey@0"
I20260925 15:18:53.423619 10431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:53.443817 10431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:53.453890 10431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:53.454147 10431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:53.467278 10431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:53.467440 10431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:53.467656 10431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:53.467792 10431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:18:53 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349533, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:53.487855 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.464414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.8.226.66:46765)
0925 15:18:53.465374 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:53.465436 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:53.468464 (+ 3028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:53.469309 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:53.469321 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:53.470155 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:53.471022 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.471052 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.472186 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.472192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.474491 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.474511 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.475563 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.475570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.475813 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.476511 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:53.476536 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:53.479221 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:53.483240 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.483247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.483258 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.483868 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.484252 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.484262 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.484265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.484434 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.485002 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:53.485010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:53.485400 (+ 390us) client_negotiation.cc:769] Sending connection context
0925 15:18:53.485747 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 15:18:53.486333 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":377,"threads_started":1}
I20260925 15:18:53.488852 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.463915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:46765 (local address 127.8.226.126:34749)
0925 15:18:53.464162 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:53.464170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:53.465565 (+ 1395us) server_negotiation.cc:407] Connection header received
0925 15:18:53.468794 (+ 3229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:53.468800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:53.468920 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:53.469048 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:53.471282 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.472000 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.474705 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.475318 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.475456 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.479479 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:53.479489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:53.479496 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:53.479550 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:53.483062 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.484005 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.484010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.484014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.484134 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.484553 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.484558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.484562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.484855 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:53.485032 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:53.488441 (+ 3409us) server_negotiation.cc:299] Negotiation successful
0925 15:18:53.488642 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:18:53.511433 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.495537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.8.226.66:51489)
0925 15:18:53.496256 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:53.496283 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:53.496454 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:53.496778 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:53.496786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:53.497522 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:53.497990 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.498007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.498986 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.498994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.501137 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.501154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.502161 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.502172 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.502456 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.503261 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:53.503309 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:53.504437 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:53.508077 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.508082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.508086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.508550 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.509074 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.509081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.509086 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.509271 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.510049 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:53.510054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:53.510343 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 15:18:53.510526 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 15:18:53.510904 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":340,"threads_started":1}
I20260925 15:18:53.512162 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.495370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:51489 (local address 127.8.226.126:34749)
0925 15:18:53.495552 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:53.495557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:53.496385 (+ 828us) server_negotiation.cc:407] Connection header received
0925 15:18:53.496569 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:53.496573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:53.496629 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:53.496704 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:53.498144 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.498853 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.501367 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.501914 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.502050 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.504700 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:53.504708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:53.504715 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:53.504770 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:53.507950 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.508716 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.508725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.508732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.508909 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.509401 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.509408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.509411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.509835 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:53.509985 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:53.511531 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0925 15:18:53.511819 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:18:53.521831 10431 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:36447
I20260925 15:18:53.521852 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:36447 every 8 connection(s)
I20260925 15:18:53.522959 10431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:53.524948 10566 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34749
I20260925 15:18:53.525220 10566 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:53.525820 10566 heartbeater.cc:511] Master 127.8.226.126:34749 requested a full tablet report, sending...
I20260925 15:18:53.527115 10220 ts_manager.cc:194] Registered new tserver with Master: e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447)
I20260925 15:18:53.527963 10220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:46765
I20260925 15:18:53.532908 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 10431
I20260925 15:18:53.533146 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:18:53.533582 9097 external_mini_cluster.cc:1428] Setting key 18b11fd18a2fa25169d8709195fdf9c0
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:18:53.570237 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34749
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:53.869799 10570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:53.870069 10570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:53.870177 10570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:53.877123 10570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:53.877228 10570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:53.877318 10570 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:18:53.877404 10570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:18:53.886615 10570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:34749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:53.887795 10570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:53.889040 10570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:53.901834 10576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:53.901875 10575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:53.901859 10578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:53.902724 10570 server_base.cc:1034] running on GCE node
I20260925 15:18:53.903307 10570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:53.904165 10570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:53.905395 10570 hybrid_clock.cc:648] HybridClock initialized: now 1790349533905336 us; error 37 us; skew 500 ppm
I20260925 15:18:53.905665 10570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:53 dist-test-slave-8vw7 krb5kdc[10175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349533, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:53.908631 10570 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:18:53.910128 10570 webserver.cc:460] Webserver started at http://127.8.226.67:40509/ using document root <none> and password file <none>
I20260925 15:18:53.910786 10570 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:53.910909 10570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:53.911216 10570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:53.913938 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "fa9eb81e80264bdaa6e882724985997f"
format_stamp: "Formatted at 2026-09-25 15:18:53 on dist-test-slave-8vw7"
server_key: "9174b5a621eac92ac56b950b65fee880"
server_key_iv: "65d3ed9cae6800b9d04e7f0609079643"
server_key_version: "encryptionkey@0"
I20260925 15:18:53.914724 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "fa9eb81e80264bdaa6e882724985997f"
format_stamp: "Formatted at 2026-09-25 15:18:53 on dist-test-slave-8vw7"
server_key: "9174b5a621eac92ac56b950b65fee880"
server_key_iv: "65d3ed9cae6800b9d04e7f0609079643"
server_key_version: "encryptionkey@0"
I20260925 15:18:53.919552 10570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:18:53.922641 10585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:53.923880 10570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:18:53.924032 10570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "fa9eb81e80264bdaa6e882724985997f"
format_stamp: "Formatted at 2026-09-25 15:18:53 on dist-test-slave-8vw7"
server_key: "9174b5a621eac92ac56b950b65fee880"
server_key_iv: "65d3ed9cae6800b9d04e7f0609079643"
server_key_version: "encryptionkey@0"
I20260925 15:18:53.924240 10570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:53.958386 10570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:53.962255 10570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:53.962512 10570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:53.965214 10570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:53.965323 10570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:53.965464 10570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:53.965529 10570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:18:53 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349533, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:53.990370 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.966024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:36983 (local address 127.8.226.126:34749)
0925 15:18:53.966247 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:53.966252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:53.967903 (+ 1651us) server_negotiation.cc:407] Connection header received
0925 15:18:53.969958 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:53.969964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:53.970060 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:53.970182 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:53.972596 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.973966 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.977152 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.977674 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.977812 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.983124 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:53.983134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:53.983141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:53.983185 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:53.986365 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.987452 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.987456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.987458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.987585 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:53.988150 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:53.988153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:53.988155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:53.988446 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:53.988604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:53.989647 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0925 15:18:53.989917 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 15:18:53.992296 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.966929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.8.226.67:36983)
0925 15:18:53.967742 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:53.967783 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:53.969691 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:53.970452 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:53.970467 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:53.971405 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:53.972376 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.972413 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.974185 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.974193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.976914 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:53.976936 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:53.977986 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:53.977996 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:53.978339 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:53.979204 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:53.979248 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:53.982857 (+ 3609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:53.986533 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.986546 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.986566 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.987314 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.987699 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:53.987707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:53.987713 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:53.988026 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:53.988582 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:53.988593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:53.989129 (+ 536us) client_negotiation.cc:769] Sending connection context
0925 15:18:53.989519 (+ 390us) client_negotiation.cc:240] Negotiation successful
0925 15:18:53.990303 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":377,"threads_started":1}
I20260925 15:18:54.013533 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.998861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:46955 (local address 127.8.226.126:34749)
0925 15:18:53.999060 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:53.999066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:53.999871 (+ 805us) server_negotiation.cc:407] Connection header received
0925 15:18:54.000078 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.000086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.000175 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.000293 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:54.001597 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.002588 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.004440 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.004859 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.005021 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.007512 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:54.007516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:54.007520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:54.007547 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:54.010952 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.011577 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.011582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.011586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.011780 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.012155 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.012162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.012165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.012404 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:54.012549 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.012919 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.013155 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:18:54.013641 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:53.999095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.8.226.67:46955)
0925 15:18:53.999753 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:53.999776 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:53.999930 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.000333 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.000341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.000987 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:54.001453 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.001464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.002709 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.002716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.004285 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.004296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.005081 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.005087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.005319 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.006007 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:54.006042 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:54.007284 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:54.011075 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:54.011085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:54.011090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:54.011459 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:54.011915 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:54.011920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:54.011924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:54.012050 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:54.012515 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:54.012519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:54.012688 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.012824 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.013193 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":348,"threads_started":1}
I20260925 15:18:54.016719 10570 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:37591
I20260925 15:18:54.017374 10704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:37591 every 8 connection(s)
I20260925 15:18:54.018241 10570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:54.020129 10705 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34749
I20260925 15:18:54.020354 10705 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:54.020896 10705 heartbeater.cc:511] Master 127.8.226.126:34749 requested a full tablet report, sending...
I20260925 15:18:54.022097 10220 ts_manager.cc:194] Registered new tserver with Master: fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591)
I20260925 15:18:54.022578 10427 heartbeater.cc:503] Master 127.8.226.126:34749 was elected leader, sending a full tablet report...
I20260925 15:18:54.022935 10220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:36983
I20260925 15:18:54.025090 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 10570
I20260925 15:18:54.025239 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:18:54.025594 9097 external_mini_cluster.cc:1428] Setting key bb5e9f8c0bc0e300ef41bf214fd4c2aa
I20260925 15:18:54.028327 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:18:54 dist-test-slave-8vw7 krb5kdc[10175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:18:54 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:54.072407 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.059031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.8.226.126:34749)
0925 15:18:54.059255 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.059263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.059479 (+ 216us) server_negotiation.cc:407] Connection header received
0925 15:18:54.059704 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.059708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.059771 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.059876 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:54.061235 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.061955 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.063247 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.063681 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.063811 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.067510 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:54.067515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:54.067518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:54.067551 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:54.070077 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.070645 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.070650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.070657 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.070775 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.071110 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.071117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.071123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.071370 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:54.071612 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.071892 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.072094 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:18:54.076407 10219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:54.083145 10219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:54.126595 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.094586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.8.226.66:36447)
0925 15:18:54.118651 (+ 24065us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.118657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.118698 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:18:54.118771 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.118776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.118945 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.119542 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.119956 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.121446 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.124312 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.125462 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.125636 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.125847 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.126110 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.126270 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23832,"thread_start_us":23711,"threads_started":1}
I20260925 15:18:54.126959 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.094362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36447 (local address 127.0.0.1:40776)
0925 15:18:54.107221 (+ 12859us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.107285 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.107507 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.119165 (+ 11658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.119169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.119194 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.119711 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.119726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.121590 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.121596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.124118 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.124132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.125662 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.125668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.125851 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.125909 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.126063 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.126253 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12548,"mutex_wait_us":46,"thread_start_us":12426,"threads_started":1}
I20260925 15:18:54.127903 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.094608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42419 (local address 127.0.0.1:59102)
0925 15:18:54.117522 (+ 22914us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.117545 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.117691 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.119094 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.119112 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.119147 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.120559 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.120571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.122323 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.122328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.124699 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.124714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.127070 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.127078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.127269 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.127313 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.127383 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.127497 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22755,"mutex_wait_us":67,"thread_start_us":25261,"threads_started":1}
I20260925 15:18:54.128516 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.098255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:37591 (local address 127.0.0.1:57084)
0925 15:18:54.120134 (+ 21879us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.120151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.120264 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.120694 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.120698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.120721 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.121082 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.121091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.122441 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.122446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.124819 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.124832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.127717 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.127722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.127903 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.127939 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.128024 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.128151 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21756,"mutex_wait_us":52,"thread_start_us":18963,"threads_started":1}
I20260925 15:18:54.128809 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.094832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.8.226.65:42419)
0925 15:18:54.118280 (+ 23448us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.118292 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.118327 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:18:54.118451 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.118463 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.118835 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.119073 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.120737 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.122180 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.124852 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.126329 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.126461 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.128286 (+ 1825us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.128351 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.128502 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23238,"thread_start_us":23093,"threads_started":1}
I20260925 15:18:54.129282 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.098692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.8.226.67:37591)
0925 15:18:54.115822 (+ 17130us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.115835 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.120263 (+ 4428us) server_negotiation.cc:407] Connection header received
0925 15:18:54.120385 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.120390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.120529 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.120712 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.121268 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.122307 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.124991 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.127253 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.127380 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.127584 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.128761 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.128922 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16875,"thread_start_us":17143,"threads_started":1}
I20260925 15:18:54.131170 10500 tablet_service.cc:1467] Processing CreateTablet for tablet 0793ece7682c47a8955eb917ba5ae0ab (DEFAULT_TABLE table=test-table [id=49635f4ce43447c8873712015da6292e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:54.131901 10635 tablet_service.cc:1467] Processing CreateTablet for tablet 0793ece7682c47a8955eb917ba5ae0ab (DEFAULT_TABLE table=test-table [id=49635f4ce43447c8873712015da6292e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:54.132579 10500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0793ece7682c47a8955eb917ba5ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.132157 10357 tablet_service.cc:1467] Processing CreateTablet for tablet 0793ece7682c47a8955eb917ba5ae0ab (DEFAULT_TABLE table=test-table [id=49635f4ce43447c8873712015da6292e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:54.133226 10635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0793ece7682c47a8955eb917ba5ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.133412 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0793ece7682c47a8955eb917ba5ae0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.144304 10724 tablet_bootstrap.cc:492] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Bootstrap starting.
I20260925 15:18:54.146432 10725 tablet_bootstrap.cc:492] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Bootstrap starting.
I20260925 15:18:54.148002 10724 tablet_bootstrap.cc:654] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.149281 10724 log.cc:824] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:54.149686 10726 tablet_bootstrap.cc:492] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Bootstrap starting.
I20260925 15:18:54.149873 10725 tablet_bootstrap.cc:654] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.151124 10725 log.cc:824] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:54.152632 10726 tablet_bootstrap.cc:654] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.153684 10726 log.cc:824] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:54.154559 10725 tablet_bootstrap.cc:492] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: No bootstrap required, opened a new log
I20260925 15:18:54.154892 10725 ts_tablet_manager.cc:1397] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:18:54.159039 10724 tablet_bootstrap.cc:492] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: No bootstrap required, opened a new log
I20260925 15:18:54.159303 10726 tablet_bootstrap.cc:492] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: No bootstrap required, opened a new log
I20260925 15:18:54.159356 10724 ts_tablet_manager.cc:1397] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260925 15:18:54.159650 10726 ts_tablet_manager.cc:1397] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260925 15:18:54.163592 10725 raft_consensus.cc:348] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.163955 10725 raft_consensus.cc:374] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.164077 10725 raft_consensus.cc:729] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3448e6707a74a34a0c2955310e8b6be, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.164666 10725 consensus_queue.cc:260] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.166379 10725 ts_tablet_manager.cc:1428] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 15:18:54.169776 10724 raft_consensus.cc:348] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.169776 10726 raft_consensus.cc:348] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.170167 10724 raft_consensus.cc:374] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.170167 10726 raft_consensus.cc:374] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.170265 10724 raft_consensus.cc:729] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9eb81e80264bdaa6e882724985997f, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.170265 10726 raft_consensus.cc:729] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bb45b03d494bdd951daaae74cbb784, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.170819 10726 consensus_queue.cc:260] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.170819 10724 consensus_queue.cc:260] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.171885 10732 raft_consensus.cc:482] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:54.172008 10705 heartbeater.cc:503] Master 127.8.226.126:34749 was elected leader, sending a full tablet report...
I20260925 15:18:54.172161 10732 raft_consensus.cc:504] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.172432 10726 ts_tablet_manager.cc:1428] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:18:54.172492 10724 ts_tablet_manager.cc:1428] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260925 15:18:54.172993 10566 heartbeater.cc:503] Master 127.8.226.126:34749 was elected leader, sending a full tablet report...
I20260925 15:18:54.173964 10732 leader_election.cc:288] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447), c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419)
I20260925 15:18:54.180634 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.175349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:32905 (local address 127.8.226.66:36447)
0925 15:18:54.175527 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.175531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.175545 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:18:54.175588 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.175591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.175640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.175734 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.176196 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.177282 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.179076 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.179801 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.179969 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.180033 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.180288 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.180399 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:18:54.180748 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.174175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36447 (local address 127.8.226.67:32905)
0925 15:18:54.174788 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.174807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.174895 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.175748 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.175752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.175776 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.176044 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.176052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.177435 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.177440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.178926 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.178938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.179983 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.179988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.180144 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.180164 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.180256 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.180365 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":2,"thread_start_us":318,"threads_started":1}
I20260925 15:18:54.181119 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.174165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42419 (local address 127.8.226.67:40485)
0925 15:18:54.174350 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.174371 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.174487 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.176758 (+ 2271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.176764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.176785 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.177050 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.177064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.178353 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.178357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.179473 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.179484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.180481 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.180487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.180671 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.180722 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.180814 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.180928 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:18:54.181231 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.176263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:40485 (local address 127.8.226.65:42419)
0925 15:18:54.176448 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.176455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.176474 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:18:54.176527 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.176532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.176613 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.176720 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.177176 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.178175 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.179630 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.180337 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.180487 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.180548 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.180912 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.181063 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:18:54.182024 10521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0793ece7682c47a8955eb917ba5ae0ab" candidate_uuid: "fa9eb81e80264bdaa6e882724985997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bb45b03d494bdd951daaae74cbb784" is_pre_election: true
I20260925 15:18:54.182490 10521 raft_consensus.cc:2454] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa9eb81e80264bdaa6e882724985997f in term 0.
I20260925 15:18:54.182446 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0793ece7682c47a8955eb917ba5ae0ab" candidate_uuid: "fa9eb81e80264bdaa6e882724985997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3448e6707a74a34a0c2955310e8b6be" is_pre_election: true
I20260925 15:18:54.182933 10381 raft_consensus.cc:2454] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa9eb81e80264bdaa6e882724985997f in term 0.
I20260925 15:18:54.183117 10587 leader_election.cc:302] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8bb45b03d494bdd951daaae74cbb784, fa9eb81e80264bdaa6e882724985997f; no voters:
I20260925 15:18:54.183446 10732 raft_consensus.cc:2781] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:54.183588 10732 raft_consensus.cc:482] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:54.183714 10732 raft_consensus.cc:3037] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.185807 10732 raft_consensus.cc:504] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.186370 10732 leader_election.cc:288] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [CANDIDATE]: Term 1 election: Requested vote from peers e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447), c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419)
I20260925 15:18:54.186698 10521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0793ece7682c47a8955eb917ba5ae0ab" candidate_uuid: "fa9eb81e80264bdaa6e882724985997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bb45b03d494bdd951daaae74cbb784"
I20260925 15:18:54.186905 10521 raft_consensus.cc:3037] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.186889 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0793ece7682c47a8955eb917ba5ae0ab" candidate_uuid: "fa9eb81e80264bdaa6e882724985997f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3448e6707a74a34a0c2955310e8b6be"
I20260925 15:18:54.187079 10381 raft_consensus.cc:3037] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.190075 10381 raft_consensus.cc:2454] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9eb81e80264bdaa6e882724985997f in term 1.
I20260925 15:18:54.190075 10521 raft_consensus.cc:2454] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9eb81e80264bdaa6e882724985997f in term 1.
I20260925 15:18:54.190411 10587 leader_election.cc:302] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8bb45b03d494bdd951daaae74cbb784, fa9eb81e80264bdaa6e882724985997f; no voters:
I20260925 15:18:54.190723 10732 raft_consensus.cc:2781] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:54.191186 10732 raft_consensus.cc:686] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 1 LEADER]: Becoming Leader. State: Replica: fa9eb81e80264bdaa6e882724985997f, State: Running, Role: LEADER
I20260925 15:18:54.191682 10732 consensus_queue.cc:237] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.196004 10220 catalog_manager.cc:5495] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f reported cstate change: term changed from 0 to 1, leader changed from <none> to fa9eb81e80264bdaa6e882724985997f (127.8.226.67). New cstate: current_term: 1 leader_uuid: "fa9eb81e80264bdaa6e882724985997f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } health_report { overall_health: HEALTHY } } }
I20260925 15:18:54.234164 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.228928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.8.226.67:37591)
0925 15:18:54.229169 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.229173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.229208 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:18:54.229481 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.229485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.229556 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.230051 (+ 495us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:54.230343 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.231011 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.232216 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.232614 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.232705 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.233165 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.233718 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.233823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.233908 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.233968 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 15:18:54.241379 10381 raft_consensus.cc:1261] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Refusing update from remote peer fa9eb81e80264bdaa6e882724985997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:54.241436 10521 raft_consensus.cc:1261] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Refusing update from remote peer fa9eb81e80264bdaa6e882724985997f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:54.242229 10734 consensus_queue.cc:1035] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:54.242537 10732 consensus_queue.cc:1035] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:54.252406 10737 mvcc.cc:205] Tried to move back new op lower bound from 7333271692244103168 to 7333271692053250048. Current Snapshot: MvccSnapshot[applied={T|T < 7333271692244103168}]
I20260925 15:18:54.254770 10739 mvcc.cc:205] Tried to move back new op lower bound from 7333271692244103168 to 7333271692053250048. Current Snapshot: MvccSnapshot[applied={T|T < 7333271692244103168}]
I20260925 15:18:54.259800 10738 mvcc.cc:205] Tried to move back new op lower bound from 7333271692244103168 to 7333271692053250048. Current Snapshot: MvccSnapshot[applied={T|T < 7333271692244103168}]
I20260925 15:18:54.269498 10219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:54.269789 10219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:54.273273 10219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: Sending DeleteTablet for 3 replicas of tablet 0793ece7682c47a8955eb917ba5ae0ab
I20260925 15:18:54.274374 10500 tablet_service.cc:1514] Processing DeleteTablet for tablet 0793ece7682c47a8955eb917ba5ae0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:54 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:40776
I20260925 15:18:54.274492 10635 tablet_service.cc:1514] Processing DeleteTablet for tablet 0793ece7682c47a8955eb917ba5ae0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:54 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:57084
I20260925 15:18:54.274791 10357 tablet_service.cc:1514] Processing DeleteTablet for tablet 0793ece7682c47a8955eb917ba5ae0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:54 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:59102
I20260925 15:18:54.278234 10219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:54.278821 10219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:54.289120 10499 tablet_service.cc:1467] Processing CreateTablet for tablet bc85e61a53334119b22e42ac4712de74 (DEFAULT_TABLE table=test-table [id=e21e88ee82844f6bafdc5a41aa55724e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:54.289362 10634 tablet_service.cc:1467] Processing CreateTablet for tablet bc85e61a53334119b22e42ac4712de74 (DEFAULT_TABLE table=test-table [id=e21e88ee82844f6bafdc5a41aa55724e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:54.289768 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc85e61a53334119b22e42ac4712de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.289647 10356 tablet_service.cc:1467] Processing CreateTablet for tablet bc85e61a53334119b22e42ac4712de74 (DEFAULT_TABLE table=test-table [id=e21e88ee82844f6bafdc5a41aa55724e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:54.290011 10634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc85e61a53334119b22e42ac4712de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.290232 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc85e61a53334119b22e42ac4712de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.290953 10750 tablet_replica.cc:330] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: stopping tablet replica
I20260925 15:18:54.291348 10750 raft_consensus.cc:2229] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:54.291867 10750 raft_consensus.cc:2258] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:54.293905 10750 ts_tablet_manager.cc:1905] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:54.296552 10724 tablet_bootstrap.cc:492] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Bootstrap starting.
I20260925 15:18:54.296634 10726 tablet_bootstrap.cc:492] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Bootstrap starting.
I20260925 15:18:54.298211 10726 tablet_bootstrap.cc:654] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.298216 10724 tablet_bootstrap.cc:654] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.303377 10748 tablet_replica.cc:330] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: stopping tablet replica
I20260925 15:18:54.304018 10748 raft_consensus.cc:2229] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:54.304627 10748 raft_consensus.cc:2258] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:54.305488 10749 tablet_replica.cc:330] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: stopping tablet replica
I20260925 15:18:54.305907 10749 raft_consensus.cc:2229] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:54.306056 10725 tablet_bootstrap.cc:492] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Bootstrap starting.
I20260925 15:18:54.306572 10749 raft_consensus.cc:2258] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:54.306809 10748 ts_tablet_manager.cc:1905] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:54.307677 10725 tablet_bootstrap.cc:654] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.308524 10749 ts_tablet_manager.cc:1905] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:54.313231 10748 ts_tablet_manager.cc:1918] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:54.313388 10748 log.cc:1196] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/0793ece7682c47a8955eb917ba5ae0ab
I20260925 15:18:54.313882 10748 ts_tablet_manager.cc:1939] T 0793ece7682c47a8955eb917ba5ae0ab P e8bb45b03d494bdd951daaae74cbb784: Deleting consensus metadata
I20260925 15:18:54.315707 10749 ts_tablet_manager.cc:1918] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:54.315694 10205 catalog_manager.cc:4841] TS e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447): tablet 0793ece7682c47a8955eb917ba5ae0ab (table test-table [id=49635f4ce43447c8873712015da6292e]) successfully deleted
I20260925 15:18:54.315966 10749 log.cc:1196] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/0793ece7682c47a8955eb917ba5ae0ab
I20260925 15:18:54.316426 10749 ts_tablet_manager.cc:1939] T 0793ece7682c47a8955eb917ba5ae0ab P fa9eb81e80264bdaa6e882724985997f: Deleting consensus metadata
I20260925 15:18:54.317901 10206 catalog_manager.cc:4841] TS fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591): tablet 0793ece7682c47a8955eb917ba5ae0ab (table test-table [id=49635f4ce43447c8873712015da6292e]) successfully deleted
I20260925 15:18:54.318876 10750 ts_tablet_manager.cc:1918] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:54.319036 10750 log.cc:1196] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/0793ece7682c47a8955eb917ba5ae0ab
I20260925 15:18:54.319331 10726 tablet_bootstrap.cc:492] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: No bootstrap required, opened a new log
I20260925 15:18:54.319520 10726 ts_tablet_manager.cc:1397] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s
I20260925 15:18:54.319761 10725 tablet_bootstrap.cc:492] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: No bootstrap required, opened a new log
I20260925 15:18:54.319905 10725 ts_tablet_manager.cc:1397] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260925 15:18:54.320631 10726 raft_consensus.cc:348] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.320725 10725 raft_consensus.cc:348] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.321017 10725 raft_consensus.cc:374] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.321056 10726 raft_consensus.cc:374] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.321123 10725 raft_consensus.cc:729] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3448e6707a74a34a0c2955310e8b6be, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.321272 10750 ts_tablet_manager.cc:1939] T 0793ece7682c47a8955eb917ba5ae0ab P c3448e6707a74a34a0c2955310e8b6be: Deleting consensus metadata
I20260925 15:18:54.321288 10726 raft_consensus.cc:729] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bb45b03d494bdd951daaae74cbb784, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.321508 10725 consensus_queue.cc:260] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.321796 10726 consensus_queue.cc:260] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.322608 10726 ts_tablet_manager.cc:1428] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:54.323204 10207 catalog_manager.cc:4841] TS c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419): tablet 0793ece7682c47a8955eb917ba5ae0ab (table test-table [id=49635f4ce43447c8873712015da6292e]) successfully deleted
I20260925 15:18:54.324043 10725 ts_tablet_manager.cc:1428] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:18:54.325491 10724 tablet_bootstrap.cc:492] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: No bootstrap required, opened a new log
I20260925 15:18:54.325642 10724 ts_tablet_manager.cc:1397] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.005s
I20260925 15:18:54.326483 10724 raft_consensus.cc:348] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.326751 10724 raft_consensus.cc:374] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.326872 10724 raft_consensus.cc:729] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9eb81e80264bdaa6e882724985997f, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.327163 10724 consensus_queue.cc:260] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.327766 10724 ts_tablet_manager.cc:1428] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:18:54.470278 10730 raft_consensus.cc:482] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:54.470518 10730 raft_consensus.cc:504] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.472302 10730 leader_election.cc:288] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447), fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591)
I20260925 15:18:54.479563 10731 raft_consensus.cc:482] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:54.479812 10731 raft_consensus.cc:504] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.480198 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.472699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:37795 (local address 127.8.226.67:37591)
0925 15:18:54.472929 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.472937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.474018 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 15:18:54.474195 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.474199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.474262 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.474353 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.474997 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.476112 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.478270 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.479043 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.479174 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.479277 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.479670 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.479864 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:18:54.480307 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.472540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36447 (local address 127.8.226.65:43405)
0925 15:18:54.473494 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.473513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.473715 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.474224 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.474229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.474251 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.474630 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.474645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.475723 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.475729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.477702 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.477715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.479145 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.479151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.479343 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.479382 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.479849 (+ 467us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.479974 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":22,"thread_start_us":659,"threads_started":1}
I20260925 15:18:54.480768 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.472750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:43405 (local address 127.8.226.66:36447)
0925 15:18:54.472922 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.472928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.473599 (+ 671us) server_negotiation.cc:407] Connection header received
0925 15:18:54.473883 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.473888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.473958 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.474019 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.474810 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.475577 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.477856 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.478936 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.479080 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.479154 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.479533 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.479671 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:18:54.481177 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.473285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:37591 (local address 127.8.226.65:37795)
0925 15:18:54.473920 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.473932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.474062 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.474379 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.474383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.474407 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.474816 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.474827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.476286 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.476291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.478050 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.478058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.479334 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.479340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.479509 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.479536 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.480586 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.480702 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1,"thread_start_us":373,"threads_started":1}
I20260925 15:18:54.481380 10521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85e61a53334119b22e42ac4712de74" candidate_uuid: "c3448e6707a74a34a0c2955310e8b6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bb45b03d494bdd951daaae74cbb784" is_pre_election: true
I20260925 15:18:54.481485 10731 leader_election.cc:288] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419), fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591)
I20260925 15:18:54.481602 10521 raft_consensus.cc:2454] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3448e6707a74a34a0c2955310e8b6be in term 0.
I20260925 15:18:54.482107 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85e61a53334119b22e42ac4712de74" candidate_uuid: "c3448e6707a74a34a0c2955310e8b6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9eb81e80264bdaa6e882724985997f" is_pre_election: true
I20260925 15:18:54.482342 10309 leader_election.cc:302] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3448e6707a74a34a0c2955310e8b6be, e8bb45b03d494bdd951daaae74cbb784; no voters:
I20260925 15:18:54.482556 10655 raft_consensus.cc:2454] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3448e6707a74a34a0c2955310e8b6be in term 0.
I20260925 15:18:54.482825 10730 raft_consensus.cc:2781] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:54.482932 10730 raft_consensus.cc:482] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:54.483001 10730 raft_consensus.cc:3037] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.485886 10730 raft_consensus.cc:504] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.487262 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85e61a53334119b22e42ac4712de74" candidate_uuid: "c3448e6707a74a34a0c2955310e8b6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9eb81e80264bdaa6e882724985997f"
I20260925 15:18:54.487485 10655 raft_consensus.cc:3037] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.489926 10521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85e61a53334119b22e42ac4712de74" candidate_uuid: "c3448e6707a74a34a0c2955310e8b6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bb45b03d494bdd951daaae74cbb784"
I20260925 15:18:54.490130 10521 raft_consensus.cc:3037] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.493197 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.482121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:34671 (local address 127.8.226.65:42419)
0925 15:18:54.482715 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.482720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.483527 (+ 807us) server_negotiation.cc:407] Connection header received
0925 15:18:54.483783 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.483790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.483882 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.483955 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.484805 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.486187 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.489827 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.491449 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.491641 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.491743 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.492647 (+ 904us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.492803 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503}
I20260925 15:18:54.493999 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.481848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42419 (local address 127.8.226.66:34671)
0925 15:18:54.483423 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.483444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.483630 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.484159 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.484166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.484195 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.484594 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.484606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.486719 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.486727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.489612 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.489634 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.492097 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.492105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.492443 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.492492 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.493471 (+ 979us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.493641 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"threads_started":1}
I20260925 15:18:54.494678 10521 raft_consensus.cc:2454] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3448e6707a74a34a0c2955310e8b6be in term 1.
I20260925 15:18:54.494645 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85e61a53334119b22e42ac4712de74" candidate_uuid: "e8bb45b03d494bdd951daaae74cbb784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3448e6707a74a34a0c2955310e8b6be" is_pre_election: true
I20260925 15:18:54.494918 10381 raft_consensus.cc:2379] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e8bb45b03d494bdd951daaae74cbb784 in current term 1: Already voted for candidate c3448e6707a74a34a0c2955310e8b6be in this term.
I20260925 15:18:54.495324 10730 leader_election.cc:288] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [CANDIDATE]: Term 1 election: Requested vote from peers e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447), fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591)
I20260925 15:18:54.495855 10309 leader_election.cc:302] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3448e6707a74a34a0c2955310e8b6be, e8bb45b03d494bdd951daaae74cbb784; no voters:
I20260925 15:18:54.496160 10730 raft_consensus.cc:2781] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:54.496464 10655 raft_consensus.cc:2454] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3448e6707a74a34a0c2955310e8b6be in term 1.
I20260925 15:18:54.497332 10730 raft_consensus.cc:686] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 1 LEADER]: Becoming Leader. State: Replica: c3448e6707a74a34a0c2955310e8b6be, State: Running, Role: LEADER
I20260925 15:18:54.497763 10730 consensus_queue.cc:237] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.501458 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.483004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:50217 (local address 127.8.226.67:37591)
0925 15:18:54.494563 (+ 11559us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.494863 (+ 300us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.494886 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:18:54.494986 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.494990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.495098 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.495177 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:54.496067 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.497200 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.499124 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.500527 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.500718 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.500807 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.501070 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.501207 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11448}
I20260925 15:18:54.501477 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.482304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:37591 (local address 127.8.226.66:50217)
0925 15:18:54.493706 (+ 11402us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.493723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.493846 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.495541 (+ 1695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.495548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.495576 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:54.495912 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.495922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.497380 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.497386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.498980 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.498992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.500742 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.500748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.500904 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.500925 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.501048 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.501176 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11283,"thread_start_us":862,"threads_started":1}
I20260925 15:18:54.501471 10218 catalog_manager.cc:5495] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be reported cstate change: term changed from 0 to 1, leader changed from <none> to c3448e6707a74a34a0c2955310e8b6be (127.8.226.65). New cstate: current_term: 1 leader_uuid: "c3448e6707a74a34a0c2955310e8b6be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } health_report { overall_health: UNKNOWN } } }
I20260925 15:18:54.502168 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc85e61a53334119b22e42ac4712de74" candidate_uuid: "e8bb45b03d494bdd951daaae74cbb784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9eb81e80264bdaa6e882724985997f" is_pre_election: true
I20260925 15:18:54.502449 10655 raft_consensus.cc:2379] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e8bb45b03d494bdd951daaae74cbb784 in current term 1: Already voted for candidate c3448e6707a74a34a0c2955310e8b6be in this term.
I20260925 15:18:54.503021 10449 leader_election.cc:302] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8bb45b03d494bdd951daaae74cbb784; no voters: c3448e6707a74a34a0c2955310e8b6be, fa9eb81e80264bdaa6e882724985997f
I20260925 15:18:54.503424 10731 raft_consensus.cc:2726] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 15:18:54.517629 10428 tablet.cc:2369] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:54.520522 10706 tablet.cc:2369] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:54.525818 10567 tablet.cc:2369] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:18:54 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349532, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:54.588901 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.575139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.8.226.126:34749)
0925 15:18:54.575989 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.575994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.576018 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:18:54.576054 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.576056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.576112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.576253 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:54.577441 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.578263 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.579506 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.579892 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.580035 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.583615 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:54.583619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:54.583623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:54.583648 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:54.586051 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.586956 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.586960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.586962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.587068 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.587489 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.587492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.587494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.587733 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:54.587832 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.588322 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.588638 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":730,"thread_start_us":597,"threads_started":1}
I20260925 15:18:54.589033 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.575048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34749 (local address 127.0.0.1:39392)
0925 15:18:54.575668 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.575692 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.575863 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.576274 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.576278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.577015 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:54.577277 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.577286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.578385 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.578393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.579371 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.579381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.580076 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.580083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.580250 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.580894 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:54.580913 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:54.583412 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:54.586225 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:54.586238 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:54.586245 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:54.586840 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:54.587184 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:54.587191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:54.587197 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:54.587387 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:54.587864 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:54.587873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:54.588069 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.588256 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.588524 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260925 15:18:54.594491 10218 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:39392:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:18:54.595794 10218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:54.602631 10356 tablet_service.cc:1467] Processing CreateTablet for tablet 7685561ade0b4d8895d60175cdbf4a1c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f718e699057f4b75aa8ea8ef8fbcebd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:18:54.603169 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7685561ade0b4d8895d60175cdbf4a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.603089 10635 tablet_service.cc:1467] Processing CreateTablet for tablet 7685561ade0b4d8895d60175cdbf4a1c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f718e699057f4b75aa8ea8ef8fbcebd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:18:54.603600 10635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7685561ade0b4d8895d60175cdbf4a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.606002 10500 tablet_service.cc:1467] Processing CreateTablet for tablet 7685561ade0b4d8895d60175cdbf4a1c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f718e699057f4b75aa8ea8ef8fbcebd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:18:54.606489 10500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7685561ade0b4d8895d60175cdbf4a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:54.610014 10725 tablet_bootstrap.cc:492] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: Bootstrap starting.
I20260925 15:18:54.611001 10724 tablet_bootstrap.cc:492] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: Bootstrap starting.
I20260925 15:18:54.611543 10725 tablet_bootstrap.cc:654] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.612879 10724 tablet_bootstrap.cc:654] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.617038 10726 tablet_bootstrap.cc:492] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: Bootstrap starting.
I20260925 15:18:54.618752 10726 tablet_bootstrap.cc:654] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:54.622524 10726 tablet_bootstrap.cc:492] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: No bootstrap required, opened a new log
I20260925 15:18:54.622681 10726 ts_tablet_manager.cc:1397] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 15:18:54.623036 10724 tablet_bootstrap.cc:492] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: No bootstrap required, opened a new log
I20260925 15:18:54.623291 10724 ts_tablet_manager.cc:1397] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260925 15:18:54.623413 10726 raft_consensus.cc:348] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.623656 10726 raft_consensus.cc:374] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.623740 10726 raft_consensus.cc:729] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bb45b03d494bdd951daaae74cbb784, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.624007 10726 consensus_queue.cc:260] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.624378 10724 raft_consensus.cc:348] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.624430 10731 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.624668 10731 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.624779 10724 raft_consensus.cc:374] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.624914 10724 raft_consensus.cc:729] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9eb81e80264bdaa6e882724985997f, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.625252 10724 consensus_queue.cc:260] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.625715 10734 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.626027 10734 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.627914 10726 ts_tablet_manager.cc:1428] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:18:54.628445 10725 tablet_bootstrap.cc:492] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: No bootstrap required, opened a new log
I20260925 15:18:54.628605 10725 ts_tablet_manager.cc:1397] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.012s
I20260925 15:18:54.629338 10724 ts_tablet_manager.cc:1428] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 15:18:54.629335 10725 raft_consensus.cc:348] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.629566 10725 raft_consensus.cc:374] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:54.629688 10725 raft_consensus.cc:729] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3448e6707a74a34a0c2955310e8b6be, State: Initialized, Role: FOLLOWER
I20260925 15:18:54.629930 10725 consensus_queue.cc:260] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.630364 10730 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.630683 10730 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.631134 10725 ts_tablet_manager.cc:1428] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:54.745430 10731 raft_consensus.cc:482] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:54.745644 10731 raft_consensus.cc:504] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.746308 10731 leader_election.cc:288] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419), fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591)
I20260925 15:18:54.746845 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7685561ade0b4d8895d60175cdbf4a1c" candidate_uuid: "e8bb45b03d494bdd951daaae74cbb784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9eb81e80264bdaa6e882724985997f" is_pre_election: true
I20260925 15:18:54.746891 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7685561ade0b4d8895d60175cdbf4a1c" candidate_uuid: "e8bb45b03d494bdd951daaae74cbb784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3448e6707a74a34a0c2955310e8b6be" is_pre_election: true
I20260925 15:18:54.747138 10655 raft_consensus.cc:2454] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8bb45b03d494bdd951daaae74cbb784 in term 0.
I20260925 15:18:54.747148 10381 raft_consensus.cc:2454] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8bb45b03d494bdd951daaae74cbb784 in term 0.
I20260925 15:18:54.747597 10449 leader_election.cc:302] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c3448e6707a74a34a0c2955310e8b6be, e8bb45b03d494bdd951daaae74cbb784, fa9eb81e80264bdaa6e882724985997f; no voters:
I20260925 15:18:54.747848 10731 raft_consensus.cc:2781] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:54.747941 10731 raft_consensus.cc:482] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:54.747999 10731 raft_consensus.cc:3037] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.750378 10731 raft_consensus.cc:504] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.750885 10731 leader_election.cc:288] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [CANDIDATE]: Term 1 election: Requested vote from peers c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419), fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591)
I20260925 15:18:54.751286 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7685561ade0b4d8895d60175cdbf4a1c" candidate_uuid: "e8bb45b03d494bdd951daaae74cbb784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3448e6707a74a34a0c2955310e8b6be"
I20260925 15:18:54.751353 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7685561ade0b4d8895d60175cdbf4a1c" candidate_uuid: "e8bb45b03d494bdd951daaae74cbb784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9eb81e80264bdaa6e882724985997f"
I20260925 15:18:54.751473 10381 raft_consensus.cc:3037] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.751565 10655 raft_consensus.cc:3037] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:54.754387 10655 raft_consensus.cc:2454] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8bb45b03d494bdd951daaae74cbb784 in term 1.
I20260925 15:18:54.754402 10381 raft_consensus.cc:2454] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8bb45b03d494bdd951daaae74cbb784 in term 1.
I20260925 15:18:54.754848 10449 leader_election.cc:302] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8bb45b03d494bdd951daaae74cbb784, fa9eb81e80264bdaa6e882724985997f; no voters:
I20260925 15:18:54.755138 10731 raft_consensus.cc:2781] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:54.755461 10731 raft_consensus.cc:686] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [term 1 LEADER]: Becoming Leader. State: Replica: e8bb45b03d494bdd951daaae74cbb784, State: Running, Role: LEADER
I20260925 15:18:54.755872 10731 consensus_queue.cc:237] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } }
I20260925 15:18:54.757827 10770 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: TxnStatusTablet state changed. Reason: New leader e8bb45b03d494bdd951daaae74cbb784. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.758177 10770 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:18:54.759135 10772 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:18:54.759362 10772 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:18:54.759801 10218 catalog_manager.cc:5495] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } health_report { overall_health: UNKNOWN } } }
I20260925 15:18:54.777872 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.772602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.8.226.66:36447)
0925 15:18:54.772748 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.772753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.772856 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:18:54.773165 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.773169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.773247 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.773380 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:54.773698 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.774386 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.775646 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.776093 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.776278 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.776722 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.777299 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.777458 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.777533 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.777603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:18:54.777902 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.772545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36447 (local address 127.0.0.1:40790)
0925 15:18:54.772755 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.772773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.772933 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.773395 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.773399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.773409 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:18:54.773580 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.773587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.774517 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.774524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.775508 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.775518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.776302 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.776308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.776477 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.776523 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.777420 (+ 897us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.777438 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.777515 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.777645 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:18:54.781970 10381 raft_consensus.cc:1261] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Refusing update from remote peer e8bb45b03d494bdd951daaae74cbb784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:54.782012 10655 raft_consensus.cc:1261] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Refusing update from remote peer e8bb45b03d494bdd951daaae74cbb784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:54.782582 10731 consensus_queue.cc:1035] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:54.782848 10770 consensus_queue.cc:1035] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:54.786543 10734 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: TxnStatusTablet state changed. Reason: New leader e8bb45b03d494bdd951daaae74cbb784. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.786897 10734 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.787628 10730 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: TxnStatusTablet state changed. Reason: New leader e8bb45b03d494bdd951daaae74cbb784. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.787923 10730 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.792552 10730 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.793231 10730 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P c3448e6707a74a34a0c2955310e8b6be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.797421 10770 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.797717 10770 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:18:54.797490 10780 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.797866 10780 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P e8bb45b03d494bdd951daaae74cbb784: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:18:54.800707 10734 tablet_replica.cc:439] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8bb45b03d494bdd951daaae74cbb784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3448e6707a74a34a0c2955310e8b6be" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42419 } } peers { permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 } } peers { permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 } } }
I20260925 15:18:54.801003 10734 tablet_replica.cc:442] T 7685561ade0b4d8895d60175cdbf4a1c P fa9eb81e80264bdaa6e882724985997f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:54.811074 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.805107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.8.226.65:42419)
0925 15:18:54.805304 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.805310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.805666 (+ 356us) server_negotiation.cc:407] Connection header received
0925 15:18:54.805857 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.805861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.805927 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.806046 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:54.806576 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.807299 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.809115 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.809518 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.809617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.810065 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.810533 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.810638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.810763 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.810832 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:18:54.825650 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.820547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:55223 (local address 127.8.226.66:36447)
0925 15:18:54.820707 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.820730 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.821165 (+ 435us) server_negotiation.cc:407] Connection header received
0925 15:18:54.821355 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.821358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.821417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.821501 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:54.821898 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.822603 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.823901 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.824395 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.824540 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.824927 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.825065 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.825157 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.825295 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.825371 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:18:54.825724 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.820551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36447 (local address 127.8.226.65:55223)
0925 15:18:54.821077 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.821092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.821227 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.821520 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.821524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.821534 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:18:54.821770 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.821777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.822745 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.822752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.823751 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.823762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.824572 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.824579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.824704 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.824748 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.825158 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.825169 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.825266 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.825388 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":261,"threads_started":1}
I20260925 15:18:54.835064 10655 raft_consensus.cc:1261] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Refusing update from remote peer c3448e6707a74a34a0c2955310e8b6be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:54.835064 10521 raft_consensus.cc:1261] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Refusing update from remote peer c3448e6707a74a34a0c2955310e8b6be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:54.835675 10730 consensus_queue.cc:1035] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bb45b03d494bdd951daaae74cbb784" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:54.836242 10758 consensus_queue.cc:1035] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9eb81e80264bdaa6e882724985997f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:54.877835 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.870379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:58013 (local address 127.8.226.65:42419)
0925 15:18:54.870540 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.870545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.871315 (+ 770us) server_negotiation.cc:407] Connection header received
0925 15:18:54.871826 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.871832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.871914 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.872007 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:54.872805 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.874096 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.875667 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.876374 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.876505 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.877220 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.877339 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.877423 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.877560 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.877618 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:18:54.878119 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.870526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42419 (local address 127.8.226.66:58013)
0925 15:18:54.871207 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:54.871227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:54.871677 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:54.872035 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:54.872039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:54.872051 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:18:54.872615 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.872629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.874247 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.874251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.875530 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:54.875544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.876570 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.876577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:54.876771 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.876841 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:18:54.877448 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:18:54.877460 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:18:54.877550 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:18:54.877699 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":287,"threads_started":1}
I20260925 15:18:54.915419 10218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:54.915714 10218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:54.919030 10218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5644564aecc49e58b5dcafd171102cf: Sending DeleteTablet for 3 replicas of tablet bc85e61a53334119b22e42ac4712de74
I20260925 15:18:54.919965 10635 tablet_service.cc:1514] Processing DeleteTablet for tablet bc85e61a53334119b22e42ac4712de74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:54 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:57084
I20260925 15:18:54.921235 10356 tablet_service.cc:1514] Processing DeleteTablet for tablet bc85e61a53334119b22e42ac4712de74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:54 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:59102
I20260925 15:18:54.922358 10500 tablet_service.cc:1514] Processing DeleteTablet for tablet bc85e61a53334119b22e42ac4712de74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:54 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:40776
I20260925 15:18:54.925407 10798 tablet_replica.cc:330] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: stopping tablet replica
I20260925 15:18:54.925652 10798 raft_consensus.cc:2229] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:54.926326 10798 raft_consensus.cc:2258] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:54.933853 10798 ts_tablet_manager.cc:1905] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:54.935742 10796 tablet_replica.cc:330] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: stopping tablet replica
I20260925 15:18:54.935951 10796 raft_consensus.cc:2229] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:54.936172 10796 raft_consensus.cc:2258] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:54.937160 10796 ts_tablet_manager.cc:1905] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:54.943112 10798 ts_tablet_manager.cc:1918] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:18:54.943252 10798 log.cc:1196] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/bc85e61a53334119b22e42ac4712de74
I20260925 15:18:54.943763 10798 ts_tablet_manager.cc:1939] T bc85e61a53334119b22e42ac4712de74 P c3448e6707a74a34a0c2955310e8b6be: Deleting consensus metadata
I20260925 15:18:54.944027 10796 ts_tablet_manager.cc:1918] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:18:54.944176 10796 log.cc:1196] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/bc85e61a53334119b22e42ac4712de74
I20260925 15:18:54.944670 10796 ts_tablet_manager.cc:1939] T bc85e61a53334119b22e42ac4712de74 P fa9eb81e80264bdaa6e882724985997f: Deleting consensus metadata
I20260925 15:18:54.944986 10207 catalog_manager.cc:4841] TS c3448e6707a74a34a0c2955310e8b6be (127.8.226.65:42419): tablet bc85e61a53334119b22e42ac4712de74 (table test-table [id=e21e88ee82844f6bafdc5a41aa55724e]) successfully deleted
I20260925 15:18:54.945680 10206 catalog_manager.cc:4841] TS fa9eb81e80264bdaa6e882724985997f (127.8.226.67:37591): tablet bc85e61a53334119b22e42ac4712de74 (table test-table [id=e21e88ee82844f6bafdc5a41aa55724e]) successfully deleted
I20260925 15:18:54.947543 10799 tablet_replica.cc:330] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: stopping tablet replica
I20260925 15:18:54.947758 10799 raft_consensus.cc:2229] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:54.948022 10799 raft_consensus.cc:2258] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:54.949220 10799 ts_tablet_manager.cc:1905] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:54.957196 10799 ts_tablet_manager.cc:1918] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:18:54.957388 10799 log.cc:1196] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/bc85e61a53334119b22e42ac4712de74
I20260925 15:18:54.957942 10799 ts_tablet_manager.cc:1939] T bc85e61a53334119b22e42ac4712de74 P e8bb45b03d494bdd951daaae74cbb784: Deleting consensus metadata
I20260925 15:18:54.959110 10205 catalog_manager.cc:4841] TS e8bb45b03d494bdd951daaae74cbb784 (127.8.226.66:36447): tablet bc85e61a53334119b22e42ac4712de74 (table test-table [id=e21e88ee82844f6bafdc5a41aa55724e]) successfully deleted
Sep 25 15:18:54 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:18:54.978024 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.953955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.8.226.65:42419)
0925 15:18:54.954151 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.954157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.959505 (+ 5348us) server_negotiation.cc:407] Connection header received
0925 15:18:54.959800 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.959806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.959885 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.959991 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:54.961933 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.963101 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.964976 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.965673 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.965805 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.970806 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:54.970824 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:54.970842 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:54.970888 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:54.974869 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.975670 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.975680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.975683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.975847 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.976288 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.976293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.976297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.976734 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:54.977058 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.977363 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.977579 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:18:54.978910 10422 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59134
I20260925 15:18:54.998202 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:54.981818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.8.226.126:34749)
0925 15:18:54.982013 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:54.982019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:54.982799 (+ 780us) server_negotiation.cc:407] Connection header received
0925 15:18:54.982945 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:54.982951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:54.983030 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:54.983130 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:54.984712 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.985889 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.987762 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:54.988389 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:54.988507 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:54.991174 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:54.991182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:54.991188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:54.991246 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:54.994643 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.995907 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.995913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.995916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.996053 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:54.996469 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:54.996474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:54.996478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:54.996823 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:54.997100 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:54.997625 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 15:18:54.997862 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:18:54.998749 10218 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39400
I20260925 15:18:55.001042 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10292
I20260925 15:18:55.019342 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:55.003223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.8.226.65:42419)
0925 15:18:55.003392 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:55.003397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:55.003415 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:18:55.003469 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:55.003474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:55.003543 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:55.003664 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:55.005181 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:55.006389 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:55.008190 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:55.008896 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:55.009076 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:55.011684 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:55.011705 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:55.011710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:55.011747 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:55.015754 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:55.016535 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:55.016541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:55.016545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:55.016714 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:55.017163 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:55.017168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:55.017172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:55.017487 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:55.017739 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:55.018750 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0925 15:18:55.018973 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:18:55.019980 10422 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59140
W20260925 15:18:55.020462 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10292: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:55.020649 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 10292
I20260925 15:18:55.047556 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10431
Sep 25 15:18:55 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:18:55.064394 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:55.048525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.8.226.66:36447)
0925 15:18:55.048708 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:55.048713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:55.048731 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:18:55.048791 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:55.048796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:55.048875 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:55.049001 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:55.050515 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:55.051621 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:55.053505 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:55.053921 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:55.054011 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:55.059023 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:55.059033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:55.059047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:55.059085 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:55.061895 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:55.062594 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:55.062600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:55.062603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:55.062737 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:55.063079 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:55.063081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:55.063083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:55.063376 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:55.063608 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:55.063891 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:18:55.064039 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:18:55.065047 10561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40802
W20260925 15:18:55.065650 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10431: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:55.065820 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 10431
I20260925 15:18:55.083343 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10570
Sep 25 15:18:55 dist-test-slave-8vw7 krb5kdc[10175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:18:55.097404 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:55.083984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.8.226.67:37591)
0925 15:18:55.084466 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:55.084470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:55.084484 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:18:55.084534 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:55.084538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:55.084590 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:55.084702 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:55.085854 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:55.086587 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:55.087722 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:55.088086 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:55.088179 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:55.092293 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:55.092305 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:55.092337 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:55.092369 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:55.095245 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:55.095783 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:55.095790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:55.095792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:55.095902 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:55.096175 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:55.096178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:55.096180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:55.096487 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:55.096728 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:55.096889 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:18:55.097129 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
W20260925 15:18:55.098078 10700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57094
W20260925 15:18:55.098691 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10570: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:55.098810 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 10570
I20260925 15:18:55.118248 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10188
I20260925 15:18:55.119251 10260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:55.229313 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 10188
2026-09-25T15:18:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4360 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:55 dist-test-slave-8vw7 krb5kdc[10808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:55 dist-test-slave-8vw7 krb5kdc[10808](info): set up 2 sockets
Sep 25 15:18:55 dist-test-slave-8vw7 krb5kdc[10808](info): commencing operation
krb5kdc: starting...
W20260925 15:18:56.253311 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:18:56 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:18:56.423817 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44203
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:56.722918 10821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:56.723196 10821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:56.723284 10821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:56.730434 10821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:56.730542 10821 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:18:56.730581 10821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:56.730638 10821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:56.730693 10821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:56.730752 10821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:56.739957 10821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:44203
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:56.741080 10821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:56.742317 10821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:56.753919 10826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:56.753943 10827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:56.753903 10829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:56.754717 10821 server_base.cc:1034] running on GCE node
I20260925 15:18:56.755363 10821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:56.756577 10821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:56.757784 10821 hybrid_clock.cc:648] HybridClock initialized: now 1790349536757729 us; error 27 us; skew 500 ppm
I20260925 15:18:56.758096 10821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:56 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349536, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:56.761194 10821 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:18:56.762485 10821 webserver.cc:460] Webserver started at http://127.8.226.126:38117/ using document root <none> and password file <none>
I20260925 15:18:56.763134 10821 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:56.763262 10821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:56.763553 10821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:56.766395 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "f4655c270c254d20bb54a90e339e2509"
format_stamp: "Formatted at 2026-09-25 15:18:56 on dist-test-slave-8vw7"
server_key: "7123a74904fb69036fd26f74233d8644"
server_key_iv: "c2ae3e0d598612ccd1877891fff7213d"
server_key_version: "encryptionkey@0"
I20260925 15:18:56.767129 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "f4655c270c254d20bb54a90e339e2509"
format_stamp: "Formatted at 2026-09-25 15:18:56 on dist-test-slave-8vw7"
server_key: "7123a74904fb69036fd26f74233d8644"
server_key_iv: "c2ae3e0d598612ccd1877891fff7213d"
server_key_version: "encryptionkey@0"
I20260925 15:18:56.771965 10821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:18:56.774981 10836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:56.776065 10821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:18:56.776233 10821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "f4655c270c254d20bb54a90e339e2509"
format_stamp: "Formatted at 2026-09-25 15:18:56 on dist-test-slave-8vw7"
server_key: "7123a74904fb69036fd26f74233d8644"
server_key_iv: "c2ae3e0d598612ccd1877891fff7213d"
server_key_version: "encryptionkey@0"
I20260925 15:18:56.776453 10821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:56.804626 10821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:56.808516 10821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:56.808772 10821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:56.828878 10821 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:44203
I20260925 15:18:56.828897 10897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:44203 every 8 connection(s)
I20260925 15:18:56.830221 10821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:56.833639 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:56.839140 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 10821
I20260925 15:18:56.839365 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:18:56.839954 9097 external_mini_cluster.cc:1428] Setting key 5b098d632ed1432945f8455e0917ac6e
I20260925 15:18:56.844431 10898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Bootstrap starting.
I20260925 15:18:56.848021 10898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:56.849299 10898 log.cc:824] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:56.852226 10898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: No bootstrap required, opened a new log
Sep 25 15:18:56 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:56.860268 10898 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4655c270c254d20bb54a90e339e2509" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44203 } }
I20260925 15:18:56.860626 10898 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:56.860749 10898 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4655c270c254d20bb54a90e339e2509, State: Initialized, Role: FOLLOWER
I20260925 15:18:56.861415 10898 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4655c270c254d20bb54a90e339e2509" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44203 } }
I20260925 15:18:56.861681 10898 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:56.861840 10898 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:56.862010 10898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:56.862726 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:56.841871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.8.226.126:44203)
0925 15:18:56.842666 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:56.842681 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:56.842725 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:18:56.844883 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:56.844914 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:56.845390 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:56.845772 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:56.846964 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:56.849010 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:56.850590 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:56.851612 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:56.851792 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:56.855757 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:56.855767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:56.855780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:56.855810 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:56.858765 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:56.859485 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:56.859490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:56.859495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:56.859628 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:56.859992 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:56.859994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:56.859996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:56.860442 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:56.860749 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:56.861239 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 15:18:56.861665 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":340,"threads_started":1}
I20260925 15:18:56.864287 10898 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4655c270c254d20bb54a90e339e2509" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44203 } }
I20260925 15:18:56.864745 10898 leader_election.cc:302] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4655c270c254d20bb54a90e339e2509; no voters:
I20260925 15:18:56.865358 10898 leader_election.cc:288] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:56.865453 10903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:56.866106 10903 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [term 1 LEADER]: Becoming Leader. State: Replica: f4655c270c254d20bb54a90e339e2509, State: Running, Role: LEADER
I20260925 15:18:56.866653 10903 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4655c270c254d20bb54a90e339e2509" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44203 } }
I20260925 15:18:56.867429 10898 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:56.869539 10905 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4655c270c254d20bb54a90e339e2509. Latest consensus state: current_term: 1 leader_uuid: "f4655c270c254d20bb54a90e339e2509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4655c270c254d20bb54a90e339e2509" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44203 } } }
I20260925 15:18:56.869506 10904 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4655c270c254d20bb54a90e339e2509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4655c270c254d20bb54a90e339e2509" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44203 } } }
I20260925 15:18:56.870116 10905 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:56.870162 10904 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:56.871229 10910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:56.876973 10910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:56.885059 10910 catalog_manager.cc:1347] Generated new cluster ID: 9f9316aa17b9478a9c0ea832bd715a4e
I20260925 15:18:56.885157 10910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:56.905866 10910 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:56.907083 10910 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:56.923262 10910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Generated new TSK 0
I20260925 15:18:56.924067 10910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:18:56.968642 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44203
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:57.308789 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:57.309109 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:57.309209 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:57.315550 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:57.315634 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:57.315704 10924 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:18:57.315757 10924 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:18:57.315811 10924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:18:57.324477 10924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:57.325702 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:57.326939 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:57.339319 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:57.339290 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:57.339290 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:57.340140 10924 server_base.cc:1034] running on GCE node
I20260925 15:18:57.340709 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:57.341619 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:57.342825 10924 hybrid_clock.cc:648] HybridClock initialized: now 1790349537342787 us; error 51 us; skew 500 ppm
I20260925 15:18:57.343115 10924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:57 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:57.346035 10924 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:18:57.347518 10924 webserver.cc:460] Webserver started at http://127.8.226.65:39915/ using document root <none> and password file <none>
I20260925 15:18:57.348186 10924 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:57.348325 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:57.348697 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:57.351584 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "4b06dc95e81840b6aed3e2b9b1470f9a"
format_stamp: "Formatted at 2026-09-25 15:18:57 on dist-test-slave-8vw7"
server_key: "e0cc82da1da88e48a41f2bba7c21020f"
server_key_iv: "0fbf685417cc88d648f56f3068453d21"
server_key_version: "encryptionkey@0"
I20260925 15:18:57.352252 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "4b06dc95e81840b6aed3e2b9b1470f9a"
format_stamp: "Formatted at 2026-09-25 15:18:57 on dist-test-slave-8vw7"
server_key: "e0cc82da1da88e48a41f2bba7c21020f"
server_key_iv: "0fbf685417cc88d648f56f3068453d21"
server_key_version: "encryptionkey@0"
I20260925 15:18:57.357270 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:18:57.360467 10939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:57.361699 10924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:57.361871 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "4b06dc95e81840b6aed3e2b9b1470f9a"
format_stamp: "Formatted at 2026-09-25 15:18:57 on dist-test-slave-8vw7"
server_key: "e0cc82da1da88e48a41f2bba7c21020f"
server_key_iv: "0fbf685417cc88d648f56f3068453d21"
server_key_version: "encryptionkey@0"
I20260925 15:18:57.362054 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:57.396347 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:57.400066 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:57.400362 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:57.402921 10924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:57.403069 10924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:57.403191 10924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:57.403281 10924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:18:57 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:57.427241 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.403745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:57271 (local address 127.8.226.126:44203)
0925 15:18:57.404212 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:57.404218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:57.405463 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 15:18:57.408544 (+ 3081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:57.408552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:57.408641 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:57.408757 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:57.410836 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.411804 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.413885 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.414311 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.414443 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.419847 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:57.419855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:57.419863 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:57.419910 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:57.423749 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.424665 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.424670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.424673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.424793 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.425338 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.425347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.425353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.425648 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:57.425760 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:57.426737 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 15:18:57.426911 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
I20260925 15:18:57.429299 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.404427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.8.226.65:57271)
0925 15:18:57.405300 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:57.405349 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:57.408162 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:57.409041 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:57.409048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:57.409799 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:57.410671 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.410690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.411942 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.411950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.413716 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.413727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.414573 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.414587 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.414962 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.415760 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:57.415804 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:57.419569 (+ 3765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:57.423897 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.423911 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.423929 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.424500 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.424896 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.424900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.424905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.425137 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.425778 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:57.425789 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:57.426301 (+ 512us) client_negotiation.cc:769] Sending connection context
0925 15:18:57.426678 (+ 377us) client_negotiation.cc:240] Negotiation successful
0925 15:18:57.427482 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":315,"threads_started":1}
I20260925 15:18:57.449031 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.435168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.8.226.65:59795)
0925 15:18:57.435834 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:57.435858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:57.436008 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:57.436322 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:57.436328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:57.437059 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:57.437623 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.437642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.438815 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.438821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.439746 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.439755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.440375 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.440381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.440528 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.441077 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:57.441092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:57.442105 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:57.445850 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.445861 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.445868 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.446574 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.447093 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.447100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.447105 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.447279 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.447889 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:57.447893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:57.448162 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 15:18:57.448352 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 15:18:57.448644 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":339,"threads_started":1}
I20260925 15:18:57.449684 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.435170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:59795 (local address 127.8.226.126:44203)
0925 15:18:57.435386 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:57.435393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:57.435947 (+ 554us) server_negotiation.cc:407] Connection header received
0925 15:18:57.436112 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:57.436115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:57.436184 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:57.436267 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:57.437839 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.438699 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.439861 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.440208 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.440322 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.442293 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:57.442299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:57.442303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:57.442331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:57.445693 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.446774 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.446780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.446783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.446949 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.447388 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.447395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.447398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.447786 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:57.447932 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:57.449155 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0925 15:18:57.449410 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:18:57.456233 10924 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:45037
I20260925 15:18:57.456255 11059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:45037 every 8 connection(s)
I20260925 15:18:57.457523 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:57.459425 11060 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44203
I20260925 15:18:57.459635 11060 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:57.460240 11060 heartbeater.cc:511] Master 127.8.226.126:44203 requested a full tablet report, sending...
I20260925 15:18:57.462028 10853 ts_manager.cc:194] Registered new tserver with Master: 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037)
I20260925 15:18:57.463358 10853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:57271
I20260925 15:18:57.468120 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 10924
I20260925 15:18:57.468309 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:18:57.469055 9097 external_mini_cluster.cc:1428] Setting key cae6a8f03782a4628e350190560b2825
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:18:57.509518 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44203
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:57.808311 11064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:57.808643 11064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:57.808820 11064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:57.815929 11064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:57.816063 11064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:57.816222 11064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:18:57.816309 11064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:18:57.816429 11064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:18:57.825469 11064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:57.826651 11064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:57.827941 11064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:57.840758 11072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:57.840777 11070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:57.841074 11069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:57.842060 11064 server_base.cc:1034] running on GCE node
I20260925 15:18:57.842628 11064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:57.843604 11064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:57.844830 11064 hybrid_clock.cc:648] HybridClock initialized: now 1790349537844772 us; error 57 us; skew 500 ppm
I20260925 15:18:57.845124 11064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:57 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:57.848206 11064 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:18:57.849526 11064 webserver.cc:460] Webserver started at http://127.8.226.66:32997/ using document root <none> and password file <none>
I20260925 15:18:57.850129 11064 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:57.850224 11064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:57.850458 11064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:57.852953 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "6a4da162339b4049b533914899b5f859"
format_stamp: "Formatted at 2026-09-25 15:18:57 on dist-test-slave-8vw7"
server_key: "978bc184fd69bdad32beb597750242c4"
server_key_iv: "0a0fad02d73bae647f3bf04062c340da"
server_key_version: "encryptionkey@0"
I20260925 15:18:57.853665 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "6a4da162339b4049b533914899b5f859"
format_stamp: "Formatted at 2026-09-25 15:18:57 on dist-test-slave-8vw7"
server_key: "978bc184fd69bdad32beb597750242c4"
server_key_iv: "0a0fad02d73bae647f3bf04062c340da"
server_key_version: "encryptionkey@0"
I20260925 15:18:57.858170 11064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:18:57.861121 11079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:57.862275 11064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:18:57.862390 11064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "6a4da162339b4049b533914899b5f859"
format_stamp: "Formatted at 2026-09-25 15:18:57 on dist-test-slave-8vw7"
server_key: "978bc184fd69bdad32beb597750242c4"
server_key_iv: "0a0fad02d73bae647f3bf04062c340da"
server_key_version: "encryptionkey@0"
I20260925 15:18:57.862531 11064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:57.891526 11064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:57.895426 11064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:57.895630 11064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:57.898121 11064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:57.898205 11064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:57.898351 11064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:57.898465 11064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:18:57 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:57.921559 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.898585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:47337 (local address 127.8.226.126:44203)
0925 15:18:57.898778 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:57.898784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:57.900063 (+ 1279us) server_negotiation.cc:407] Connection header received
0925 15:18:57.902209 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:57.902216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:57.902327 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:57.902434 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:57.905128 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.905822 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.908928 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.909487 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.909644 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.914953 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:57.914957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:57.914961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:57.914992 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:57.917885 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.918984 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.918991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.918995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.919149 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.919579 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.919584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.919587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.919941 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:57.920091 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:57.921030 (+ 939us) server_negotiation.cc:299] Negotiation successful
0925 15:18:57.921240 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:18:57.922494 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.899092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.8.226.66:47337)
0925 15:18:57.899883 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:57.899921 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:57.901959 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:57.902855 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:57.902868 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:57.903856 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:57.904917 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.904971 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.906001 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.906009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.908774 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.908790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.909771 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.909783 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.910212 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.911036 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:57.911070 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:57.914714 (+ 3644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:57.918021 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.918033 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.918053 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.918823 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.919270 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.919276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.919279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.919460 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.920163 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:57.920170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:57.920621 (+ 451us) client_negotiation.cc:769] Sending connection context
0925 15:18:57.920917 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 15:18:57.921435 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":354,"threads_started":1}
I20260925 15:18:57.945961 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.928434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:41577 (local address 127.8.226.126:44203)
0925 15:18:57.928636 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:57.928644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:57.929488 (+ 844us) server_negotiation.cc:407] Connection header received
0925 15:18:57.929588 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:57.929608 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:57.929697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:57.929801 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:57.931372 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.932537 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.935620 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.936340 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.936441 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.939192 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:57.939201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:57.939209 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:57.939251 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:57.942870 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.943815 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.943820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.943823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.944014 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:57.944452 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:57.944457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:57.944460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:57.944848 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:57.945017 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:57.945588 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 15:18:57.945754 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:18:57.946457 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:57.928452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.8.226.66:41577)
0925 15:18:57.929393 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:57.929408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:57.929501 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:57.930006 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:57.930011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:57.930703 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:57.931186 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.931202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.932755 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.932761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.935448 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:57.935465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:57.936752 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:57.936759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:57.936946 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:57.937594 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:57.937621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:57.938952 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:57.943022 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.943025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.943028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.943682 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.944164 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:57.944173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:57.944176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:57.944345 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:57.945024 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:57.945031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:57.945327 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 15:18:57.945474 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:18:57.945808 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":647,"threads_started":1}
I20260925 15:18:57.949946 11064 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:36085
I20260925 15:18:57.950037 11198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:36085 every 8 connection(s)
I20260925 15:18:57.951197 11064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:57.952983 11199 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44203
I20260925 15:18:57.953198 11199 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:57.953769 11199 heartbeater.cc:511] Master 127.8.226.126:44203 requested a full tablet report, sending...
I20260925 15:18:57.954977 10853 ts_manager.cc:194] Registered new tserver with Master: 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085)
I20260925 15:18:57.955035 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 11064
I20260925 15:18:57.955199 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:18:57.955603 9097 external_mini_cluster.cc:1428] Setting key bda1ebaed743978718949fbd5f2868ee
I20260925 15:18:57.955826 10853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:47337
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:18:57.992075 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44203
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:58.305850 11203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:58.306100 11203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:58.306195 11203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:58.313153 11203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:58.313282 11203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:58.313401 11203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:18:58.313459 11203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:18:58.313525 11203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:18:58.322913 11203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:58.324047 11203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:58.325366 11203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:58.338027 11209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:58.338068 11211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:58.337994 11208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:58.338768 11203 server_base.cc:1034] running on GCE node
I20260925 15:18:58.339323 11203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:58.340116 11203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:58.341322 11203 hybrid_clock.cc:648] HybridClock initialized: now 1790349538341264 us; error 48 us; skew 500 ppm
I20260925 15:18:58.341609 11203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:58 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349538, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:58.344432 11203 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:18:58.345995 11203 webserver.cc:460] Webserver started at http://127.8.226.67:39829/ using document root <none> and password file <none>
I20260925 15:18:58.346575 11203 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:58.346671 11203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:58.346907 11203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:58.349625 11203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "146888b99cb44933b0a96babf2a5f757"
format_stamp: "Formatted at 2026-09-25 15:18:58 on dist-test-slave-8vw7"
server_key: "847e57d5bc81ef777e30019b4fa873ea"
server_key_iv: "ead5e1e0b259c18eb5546719ca4c126a"
server_key_version: "encryptionkey@0"
I20260925 15:18:58.350311 11203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "146888b99cb44933b0a96babf2a5f757"
format_stamp: "Formatted at 2026-09-25 15:18:58 on dist-test-slave-8vw7"
server_key: "847e57d5bc81ef777e30019b4fa873ea"
server_key_iv: "ead5e1e0b259c18eb5546719ca4c126a"
server_key_version: "encryptionkey@0"
I20260925 15:18:58.355167 11203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:18:58.358448 11218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:58.359784 11203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:18:58.359944 11203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "146888b99cb44933b0a96babf2a5f757"
format_stamp: "Formatted at 2026-09-25 15:18:58 on dist-test-slave-8vw7"
server_key: "847e57d5bc81ef777e30019b4fa873ea"
server_key_iv: "ead5e1e0b259c18eb5546719ca4c126a"
server_key_version: "encryptionkey@0"
I20260925 15:18:58.360152 11203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:58.373579 11203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:58.377532 11203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:58.377812 11203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:58.380462 11203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:58.380587 11203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:58.380733 11203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:58.380860 11203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 15:18:58 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349538, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:58.402364 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.381018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:51271 (local address 127.8.226.126:44203)
0925 15:18:58.381244 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.381253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.382506 (+ 1253us) server_negotiation.cc:407] Connection header received
0925 15:18:58.384796 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.384802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.384891 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.385063 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:58.387137 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.388168 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.390924 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.391355 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.391514 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.395716 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:58.395724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:58.395733 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:58.395763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:58.398945 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.399867 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.399874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.399880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.400049 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.400666 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.400669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.400671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.400901 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:58.401086 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.401793 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.401995 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:18:58.403366 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.381542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.8.226.67:51271)
0925 15:18:58.382342 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.382383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.384300 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.385283 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.385298 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.385951 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:58.386896 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.386927 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.388328 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.388336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.390712 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.390723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.391610 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.391617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.391840 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.392515 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:58.392538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:58.395486 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:58.399075 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:58.399085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:58.399100 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:58.399735 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:58.400198 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:58.400210 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:58.400216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:58.400536 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:58.401064 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:58.401071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:58.401384 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.401652 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.402158 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":345,"threads_started":1}
I20260925 15:18:58.423959 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.409958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.8.226.67:48073)
0925 15:18:58.410608 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.410628 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.410917 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.411266 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.411271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.411938 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:58.412429 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.412446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.414013 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.414018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.415507 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.415521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.416418 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.416426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.416607 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.417265 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:58.417281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:58.418457 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:58.421394 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:58.421398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:58.421402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:58.421775 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:58.422146 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:58.422150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:58.422154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:58.422265 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:58.422789 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:58.422793 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:58.422965 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.423132 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.423439 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":407,"threads_started":1}
I20260925 15:18:58.424072 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.409479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:48073 (local address 127.8.226.126:44203)
0925 15:18:58.409707 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.409715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.410782 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 15:18:58.411069 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.411074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.411143 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.411243 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:58.412598 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.413854 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.415672 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.416236 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.416359 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.418647 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:58.418652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:58.418662 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:58.418690 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:58.421272 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.421879 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.421885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.421889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.422021 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.422359 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.422365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.422368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.422677 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:58.422832 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.423499 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.423752 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:18:58.436746 11203 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:38689
I20260925 15:18:58.436787 11337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:38689 every 8 connection(s)
I20260925 15:18:58.438000 11203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:58.439882 11338 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44203
I20260925 15:18:58.440083 11338 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:58.440653 11338 heartbeater.cc:511] Master 127.8.226.126:44203 requested a full tablet report, sending...
I20260925 15:18:58.441946 10853 ts_manager.cc:194] Registered new tserver with Master: 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:58.442766 10853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:51271
I20260925 15:18:58.446916 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 11203
I20260925 15:18:58.447100 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:18:58.447539 9097 external_mini_cluster.cc:1428] Setting key ae547dff96abc55d541a2bb1658259c0
I20260925 15:18:58.450268 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:58.465920 11060 heartbeater.cc:503] Master 127.8.226.126:44203 was elected leader, sending a full tablet report...
Sep 25 15:18:58 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:18:58 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:58.503674 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.483070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.8.226.126:44203)
0925 15:18:58.483319 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.483325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.483541 (+ 216us) server_negotiation.cc:407] Connection header received
0925 15:18:58.483667 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.483670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.483738 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.483820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:58.485077 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.485765 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.486912 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.487306 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.487399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.498680 (+ 11281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:58.498685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:58.498695 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:58.498730 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:58.501297 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.501867 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.501874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.501881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.501990 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.502360 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.502365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.502367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.502607 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:58.502832 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.503090 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.503288 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:18:58.507659 10853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:58.514251 10853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:58.550608 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.526034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.8.226.67:38689)
0925 15:18:58.541114 (+ 15080us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.541126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.541168 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:18:58.541301 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.541312 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.541645 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.543088 (+ 1443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:58.543474 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.545058 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.547870 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.549380 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.549533 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.549799 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.550093 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.550271 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14813,"thread_start_us":15799,"threads_started":1}
I20260925 15:18:58.550644 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.525278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38689 (local address 127.0.0.1:34426)
0925 15:18:58.535663 (+ 10385us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.535680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.535813 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.542199 (+ 6386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.542216 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.542278 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:58.542815 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.542832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.545181 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.545204 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.547674 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.547691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.549610 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.549616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.549847 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.549905 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.550094 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.550283 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9923,"thread_start_us":9081,"threads_started":1}
I20260925 15:18:58.551926 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.525132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36085 (local address 127.0.0.1:53092)
0925 15:18:58.534715 (+ 9583us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.534751 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.534917 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.545402 (+ 10485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.545406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.545435 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:58.545935 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.545947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.547535 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.547542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.549404 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.549416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.551250 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.551256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.551419 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.551449 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.551512 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.551626 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9363,"thread_start_us":9925,"threads_started":1}
I20260925 15:18:58.552963 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.525306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.8.226.66:36085)
0925 15:18:58.544614 (+ 19308us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.544625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.544658 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:18:58.544784 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.544795 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.545125 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.545970 (+ 845us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:58.546199 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.547408 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.549622 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.550993 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.552269 (+ 1276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.552463 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.552542 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.552700 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19122,"thread_start_us":18951,"threads_started":1}
I20260925 15:18:58.553539 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.535622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.8.226.65:45037)
0925 15:18:58.544211 (+ 8589us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.544221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.545844 (+ 1623us) server_negotiation.cc:407] Connection header received
0925 15:18:58.546069 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.546080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.546304 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.546488 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:58.547087 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.548537 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.551492 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.552486 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.552657 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.552852 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.553103 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.553261 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8427,"thread_start_us":8316,"threads_started":1}
I20260925 15:18:58.553790 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.541206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:45037 (local address 127.0.0.1:45454)
0925 15:18:58.545674 (+ 4468us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.545724 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.545854 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.546495 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.546499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.546524 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:58.546893 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.546904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.548730 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.548735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.551351 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.551363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.552707 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.552713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.552877 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.552907 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.553228 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.553342 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4329,"thread_start_us":4136,"threads_started":1}
I20260925 15:18:58.555001 11268 tablet_service.cc:1467] Processing CreateTablet for tablet cb74c95f705949c8b670b883e327b1dd (DEFAULT_TABLE table=test-table [id=b4e0952753d443458ac97d41bfe7e877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:58.556394 11268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb74c95f705949c8b670b883e327b1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:58.556380 11129 tablet_service.cc:1467] Processing CreateTablet for tablet cb74c95f705949c8b670b883e327b1dd (DEFAULT_TABLE table=test-table [id=b4e0952753d443458ac97d41bfe7e877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:58.557219 10990 tablet_service.cc:1467] Processing CreateTablet for tablet cb74c95f705949c8b670b883e327b1dd (DEFAULT_TABLE table=test-table [id=b4e0952753d443458ac97d41bfe7e877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:58.557714 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb74c95f705949c8b670b883e327b1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:58.558413 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb74c95f705949c8b670b883e327b1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:58.567814 11357 tablet_bootstrap.cc:492] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Bootstrap starting.
I20260925 15:18:58.570928 11357 tablet_bootstrap.cc:654] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:58.571954 11357 log.cc:824] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:58.577782 11359 tablet_bootstrap.cc:492] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Bootstrap starting.
I20260925 15:18:58.578706 11358 tablet_bootstrap.cc:492] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Bootstrap starting.
I20260925 15:18:58.582864 11358 tablet_bootstrap.cc:654] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:58.583287 11359 tablet_bootstrap.cc:654] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:58.584133 11359 log.cc:824] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:58.584218 11358 log.cc:824] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:58.586941 11358 tablet_bootstrap.cc:492] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: No bootstrap required, opened a new log
I20260925 15:18:58.587136 11359 tablet_bootstrap.cc:492] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: No bootstrap required, opened a new log
I20260925 15:18:58.587231 11358 ts_tablet_manager.cc:1397] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260925 15:18:58.587340 11357 tablet_bootstrap.cc:492] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: No bootstrap required, opened a new log
I20260925 15:18:58.587389 11359 ts_tablet_manager.cc:1397] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260925 15:18:58.587563 11357 ts_tablet_manager.cc:1397] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.010s
I20260925 15:18:58.594033 11358 raft_consensus.cc:348] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.594362 11358 raft_consensus.cc:374] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:58.594456 11358 raft_consensus.cc:729] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4da162339b4049b533914899b5f859, State: Initialized, Role: FOLLOWER
I20260925 15:18:58.595036 11358 consensus_queue.cc:260] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.596000 11199 heartbeater.cc:503] Master 127.8.226.126:44203 was elected leader, sending a full tablet report...
I20260925 15:18:58.597800 11358 ts_tablet_manager.cc:1428] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260925 15:18:58.597680 11359 raft_consensus.cc:348] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.597680 11357 raft_consensus.cc:348] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.598062 11359 raft_consensus.cc:374] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:58.598060 11357 raft_consensus.cc:374] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:58.598148 11359 raft_consensus.cc:729] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Initialized, Role: FOLLOWER
I20260925 15:18:58.598148 11357 raft_consensus.cc:729] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146888b99cb44933b0a96babf2a5f757, State: Initialized, Role: FOLLOWER
I20260925 15:18:58.598680 11357 consensus_queue.cc:260] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.598680 11359 consensus_queue.cc:260] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.600291 11359 ts_tablet_manager.cc:1428] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 15:18:58.600636 11357 ts_tablet_manager.cc:1428] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:18:58.600680 11338 heartbeater.cc:503] Master 127.8.226.126:44203 was elected leader, sending a full tablet report...
I20260925 15:18:58.648176 11364 raft_consensus.cc:482] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:58.648485 11364 raft_consensus.cc:504] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.650151 11364 leader_election.cc:288] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:58.656687 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.650791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:34361 (local address 127.8.226.67:38689)
0925 15:18:58.651013 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.651018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.651446 (+ 428us) server_negotiation.cc:407] Connection header received
0925 15:18:58.651619 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.651625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.651702 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.651782 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:58.652437 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.653232 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.655036 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.655739 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.655916 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.655983 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.656280 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.656403 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:18:58.656687 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.650497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:39525 (local address 127.8.226.66:36085)
0925 15:18:58.650724 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.650747 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.651578 (+ 831us) server_negotiation.cc:407] Connection header received
0925 15:18:58.651740 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.651744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.651835 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.651925 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:58.652561 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.653336 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.654979 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.655719 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.655877 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.655965 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.656308 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.656421 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:18:58.656975 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.650403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38689 (local address 127.8.226.65:34361)
0925 15:18:58.651313 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.651337 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.651484 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.651825 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.651830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.651853 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:58.652284 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.652297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.653350 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.653354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.654874 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.654885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.655978 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.655987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.656130 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.656151 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.656221 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.656367 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":64,"thread_start_us":541,"threads_started":1}
I20260925 15:18:58.657019 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.650493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36085 (local address 127.8.226.65:39525)
0925 15:18:58.651463 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.651480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.651623 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.651950 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.651956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.651984 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:58.652398 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.652410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.653459 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.653462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.654835 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.654846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.655933 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.655939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.656090 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.656159 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.656263 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.656380 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":42,"thread_start_us":687,"threads_started":1}
I20260925 15:18:58.657914 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb74c95f705949c8b670b883e327b1dd" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859" is_pre_election: true
I20260925 15:18:58.657912 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb74c95f705949c8b670b883e327b1dd" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757" is_pre_election: true
I20260925 15:18:58.658358 11293 raft_consensus.cc:2454] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:18:58.658360 11154 raft_consensus.cc:2454] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:18:58.658978 10942 leader_election.cc:302] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146888b99cb44933b0a96babf2a5f757, 4b06dc95e81840b6aed3e2b9b1470f9a; no voters:
I20260925 15:18:58.659358 11364 raft_consensus.cc:2781] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:58.659497 11364 raft_consensus.cc:482] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:58.659591 11364 raft_consensus.cc:3037] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:58.661703 11364 raft_consensus.cc:504] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.662220 11364 leader_election.cc:288] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Requested vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:58.662559 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb74c95f705949c8b670b883e327b1dd" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859"
I20260925 15:18:58.662653 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb74c95f705949c8b670b883e327b1dd" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757"
I20260925 15:18:58.662742 11154 raft_consensus.cc:3037] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:58.662885 11293 raft_consensus.cc:3037] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:58.665094 11154 raft_consensus.cc:2454] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:18:58.665421 11293 raft_consensus.cc:2454] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:18:58.665468 10940 leader_election.cc:302] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b06dc95e81840b6aed3e2b9b1470f9a, 6a4da162339b4049b533914899b5f859; no voters:
I20260925 15:18:58.665725 11364 raft_consensus.cc:2781] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:58.666131 11364 raft_consensus.cc:686] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Becoming Leader. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Running, Role: LEADER
I20260925 15:18:58.666656 11364 consensus_queue.cc:237] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } }
I20260925 15:18:58.671206 10853 catalog_manager.cc:5495] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65). New cstate: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } health_report { overall_health: HEALTHY } } }
W20260925 15:18:58.689478 11339 tablet.cc:2369] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:58.703171 11200 tablet.cc:2369] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:58.709878 11061 tablet.cc:2369] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:18:58 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349536, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:18:58.719439 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.707305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44203 (local address 127.0.0.1:44468)
0925 15:18:58.707873 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.707888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.708004 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.708283 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.708286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.708914 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:58.709248 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.709257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.710569 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.710572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.711573 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.711584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.712208 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.712211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.712339 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.713000 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:58.713024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:58.714959 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:58.717429 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:58.717439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:58.717444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:58.717826 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:58.718137 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:58.718140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:58.718142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:58.718272 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:58.718704 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:58.718709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:58.718847 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.718966 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.719144 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":301,"threads_started":1}
I20260925 15:18:58.719514 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.707194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.8.226.126:44203)
0925 15:18:58.707381 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.707388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.707972 (+ 584us) server_negotiation.cc:407] Connection header received
0925 15:18:58.708099 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.708102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.708157 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.708240 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:58.709366 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.710452 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.711684 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.712072 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.712155 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.715119 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:58.715123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:58.715127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:58.715152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:58.717324 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.717917 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.717920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.717922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.718034 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:58.718356 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:58.718359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:58.718361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:58.718594 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:58.718718 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.719050 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.719252 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":61}
I20260925 15:18:58.723940 10853 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:44468:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:18:58.724916 10853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:58.731012 10990 tablet_service.cc:1467] Processing CreateTablet for tablet b71d50e87afe42e1bf7773a0e4314663 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7e5edfdf0c0489a946524765cc6c2bd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:18:58.730979 11129 tablet_service.cc:1467] Processing CreateTablet for tablet b71d50e87afe42e1bf7773a0e4314663 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7e5edfdf0c0489a946524765cc6c2bd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:18:58.731284 11268 tablet_service.cc:1467] Processing CreateTablet for tablet b71d50e87afe42e1bf7773a0e4314663 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7e5edfdf0c0489a946524765cc6c2bd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:18:58.731894 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71d50e87afe42e1bf7773a0e4314663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:58.731890 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71d50e87afe42e1bf7773a0e4314663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:58.732013 11268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71d50e87afe42e1bf7773a0e4314663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:58.737303 11359 tablet_bootstrap.cc:492] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: Bootstrap starting.
I20260925 15:18:58.738698 11357 tablet_bootstrap.cc:492] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: Bootstrap starting.
I20260925 15:18:58.738734 11359 tablet_bootstrap.cc:654] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:58.739656 11358 tablet_bootstrap.cc:492] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: Bootstrap starting.
I20260925 15:18:58.739897 11357 tablet_bootstrap.cc:654] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:58.740788 11358 tablet_bootstrap.cc:654] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:58.741894 11357 tablet_bootstrap.cc:492] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: No bootstrap required, opened a new log
I20260925 15:18:58.742043 11357 ts_tablet_manager.cc:1397] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:18:58.742794 11358 tablet_bootstrap.cc:492] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: No bootstrap required, opened a new log
I20260925 15:18:58.742774 11357 raft_consensus.cc:348] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.742951 11358 ts_tablet_manager.cc:1397] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:58.742965 11359 tablet_bootstrap.cc:492] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: No bootstrap required, opened a new log
I20260925 15:18:58.743013 11357 raft_consensus.cc:374] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:58.743088 11359 ts_tablet_manager.cc:1397] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 15:18:58.743108 11357 raft_consensus.cc:729] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146888b99cb44933b0a96babf2a5f757, State: Initialized, Role: FOLLOWER
I20260925 15:18:58.743338 11357 consensus_queue.cc:260] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.743661 11358 raft_consensus.cc:348] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.743886 11358 raft_consensus.cc:374] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:58.743973 11358 raft_consensus.cc:729] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4da162339b4049b533914899b5f859, State: Initialized, Role: FOLLOWER
I20260925 15:18:58.743858 11359 raft_consensus.cc:348] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.743774 11365 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.744091 11359 raft_consensus.cc:374] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:58.744124 11365 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.744182 11359 raft_consensus.cc:729] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Initialized, Role: FOLLOWER
I20260925 15:18:58.744202 11358 consensus_queue.cc:260] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.744406 11359 consensus_queue.cc:260] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.744942 11358 ts_tablet_manager.cc:1428] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:58.744807 11364 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.744752 11363 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.745080 11364 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.745152 11363 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.745563 11359 ts_tablet_manager.cc:1428] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:58.746106 11357 ts_tablet_manager.cc:1428] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 15:18:58.853983 11364 raft_consensus.cc:482] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:58.854173 11364 raft_consensus.cc:504] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.854722 11364 leader_election.cc:288] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:58.855330 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71d50e87afe42e1bf7773a0e4314663" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859" is_pre_election: true
I20260925 15:18:58.855330 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71d50e87afe42e1bf7773a0e4314663" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757" is_pre_election: true
I20260925 15:18:58.855594 11293 raft_consensus.cc:2454] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:18:58.855594 11154 raft_consensus.cc:2454] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:18:58.856052 10942 leader_election.cc:302] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 146888b99cb44933b0a96babf2a5f757, 4b06dc95e81840b6aed3e2b9b1470f9a, 6a4da162339b4049b533914899b5f859; no voters:
I20260925 15:18:58.856400 11364 raft_consensus.cc:2781] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:58.856487 11364 raft_consensus.cc:482] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:58.856616 11364 raft_consensus.cc:3037] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:58.858598 11364 raft_consensus.cc:504] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.859102 11364 leader_election.cc:288] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Requested vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:58.859552 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71d50e87afe42e1bf7773a0e4314663" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757"
I20260925 15:18:58.859552 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71d50e87afe42e1bf7773a0e4314663" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859"
I20260925 15:18:58.859797 11293 raft_consensus.cc:3037] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:58.859797 11154 raft_consensus.cc:3037] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:58.862702 11293 raft_consensus.cc:2454] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:18:58.862703 11154 raft_consensus.cc:2454] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:18:58.863124 10942 leader_election.cc:302] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146888b99cb44933b0a96babf2a5f757, 4b06dc95e81840b6aed3e2b9b1470f9a; no voters:
I20260925 15:18:58.863382 11364 raft_consensus.cc:2781] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:58.863579 11364 raft_consensus.cc:686] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Becoming Leader. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Running, Role: LEADER
I20260925 15:18:58.863808 11364 consensus_queue.cc:237] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:58.865540 11368 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: TxnStatusTablet state changed. Reason: New leader 4b06dc95e81840b6aed3e2b9b1470f9a. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.866110 11368 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:18:58.866883 11380 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:18:58.867141 11380 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:18:58.866928 10853 catalog_manager.cc:5495] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65). New cstate: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260925 15:18:58.905294 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.900032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:45037 (local address 127.0.0.1:45470)
0925 15:18:58.900333 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.900348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.900505 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.900997 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.901002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.901014 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:18:58.901209 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.901217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.902115 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.902122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.903051 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.903062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.903757 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.903763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.903940 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.903988 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:18:58.904783 (+ 795us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:18:58.904802 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.904868 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.905013 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 15:18:58.905294 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.900023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.8.226.65:45037)
0925 15:18:58.900236 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.900243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.900419 (+ 176us) server_negotiation.cc:407] Connection header received
0925 15:18:58.900714 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.900718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.900793 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.900939 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:58.901343 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.901999 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.903194 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.903601 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.903763 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.904177 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:58.904669 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:58.904849 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.904892 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.904984 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:18:58.912750 11293 raft_consensus.cc:1261] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:58.912875 11154 raft_consensus.cc:1261] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:58.913769 11368 consensus_queue.cc:1035] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:58.914067 11364 consensus_queue.cc:1035] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:58.924875 11365 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: TxnStatusTablet state changed. Reason: New leader 4b06dc95e81840b6aed3e2b9b1470f9a. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.925200 11365 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.928566 11363 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: TxnStatusTablet state changed. Reason: New leader 4b06dc95e81840b6aed3e2b9b1470f9a. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.928905 11363 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.929282 11383 mvcc.cc:205] Tried to move back new op lower bound from 7333271711377539072 to 7333271711188291584. Current Snapshot: MvccSnapshot[applied={T|T < 7333271711377539072}]
I20260925 15:18:58.931437 11385 mvcc.cc:205] Tried to move back new op lower bound from 7333271711377539072 to 7333271711188291584. Current Snapshot: MvccSnapshot[applied={T|T < 7333271711377539072}]
I20260925 15:18:58.932696 11364 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.932997 11364 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:18:58.933226 11364 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.933431 11364 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 4b06dc95e81840b6aed3e2b9b1470f9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:18:58.936834 11365 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.937080 11365 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 146888b99cb44933b0a96babf2a5f757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.938369 11363 tablet_replica.cc:439] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } } }
I20260925 15:18:58.938589 11363 tablet_replica.cc:442] T b71d50e87afe42e1bf7773a0e4314663 P 6a4da162339b4049b533914899b5f859: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:18:58.947131 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.942004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.8.226.65:45037)
0925 15:18:58.942167 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.942173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.942190 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:18:58.942355 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.942360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.942431 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.942517 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:58.942795 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.943923 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.945098 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.945811 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.945936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.946298 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:58.946440 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:58.946538 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.946723 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.946807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:18:58.963073 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.957271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:60309 (local address 127.8.226.65:45037)
0925 15:18:58.957442 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:58.957449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:58.958171 (+ 722us) server_negotiation.cc:407] Connection header received
0925 15:18:58.958346 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:58.958351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:58.958416 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:58.958497 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:58.958969 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.959715 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.961402 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.961815 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.961947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.962381 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:58.962462 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:58.962559 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:58.962676 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:18:58.962793 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":50}
I20260925 15:18:58.963335 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:58.957338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:45037 (local address 127.8.226.65:60309)
0925 15:18:58.958086 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:58.958100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:58.958240 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:58.958508 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:58.958512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:58.958520 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:18:58.958849 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.958856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.959847 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.959854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.961277 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:58.961287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:58.962002 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:58.962009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:58.962143 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:58.962186 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:18:58.962553 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:18:58.962567 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:18:58.962656 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:18:58.962828 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":23,"thread_start_us":439,"threads_started":1}
I20260925 15:18:58.973557 11293 raft_consensus.cc:1261] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:58.973742 11154 raft_consensus.cc:1261] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:58.974092 11368 consensus_queue.cc:1035] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:58.974551 11368 consensus_queue.cc:1035] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:59.033843 10853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:59.034114 10853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:59.040624 10853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Sending DeleteTablet for 3 replicas of tablet cb74c95f705949c8b670b883e327b1dd
I20260925 15:18:59.041848 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet cb74c95f705949c8b670b883e327b1dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:59 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:53092
I20260925 15:18:59.042788 11402 tablet_replica.cc:330] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: stopping tablet replica
I20260925 15:18:59.043370 11402 raft_consensus.cc:2229] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:59.043805 11402 raft_consensus.cc:2258] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:59.043824 11268 tablet_service.cc:1514] Processing DeleteTablet for tablet cb74c95f705949c8b670b883e327b1dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:59 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34426
I20260925 15:18:59.043802 10853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:59.044539 10853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:59.046128 10989 tablet_service.cc:1514] Processing DeleteTablet for tablet cb74c95f705949c8b670b883e327b1dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:59 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260925 15:18:59.046304 11403 tablet_replica.cc:330] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: stopping tablet replica
I20260925 15:18:59.046696 11403 raft_consensus.cc:2229] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:59.047076 11403 raft_consensus.cc:2258] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:59.049320 11403 ts_tablet_manager.cc:1905] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:59.052459 11402 ts_tablet_manager.cc:1905] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:59.054416 10990 tablet_service.cc:1467] Processing CreateTablet for tablet aa3a4002ee3b43558ed8a92e013aa3e6 (DEFAULT_TABLE table=test-table [id=bdf54764722a4af1b71b8c77c73565f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:59.054857 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3a4002ee3b43558ed8a92e013aa3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:59.054868 11267 tablet_service.cc:1467] Processing CreateTablet for tablet aa3a4002ee3b43558ed8a92e013aa3e6 (DEFAULT_TABLE table=test-table [id=bdf54764722a4af1b71b8c77c73565f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:59.056041 11404 tablet_replica.cc:330] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: stopping tablet replica
I20260925 15:18:59.056474 11404 raft_consensus.cc:2229] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:59.057168 11404 raft_consensus.cc:2258] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:59.058662 11129 tablet_service.cc:1467] Processing CreateTablet for tablet aa3a4002ee3b43558ed8a92e013aa3e6 (DEFAULT_TABLE table=test-table [id=bdf54764722a4af1b71b8c77c73565f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:59.059056 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3a4002ee3b43558ed8a92e013aa3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:59.061502 11359 tablet_bootstrap.cc:492] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Bootstrap starting.
I20260925 15:18:59.062017 11402 ts_tablet_manager.cc:1918] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:18:59.062233 11402 log.cc:1196] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/cb74c95f705949c8b670b883e327b1dd
I20260925 15:18:59.062873 11402 ts_tablet_manager.cc:1939] T cb74c95f705949c8b670b883e327b1dd P 6a4da162339b4049b533914899b5f859: Deleting consensus metadata
I20260925 15:18:59.063220 11359 tablet_bootstrap.cc:654] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:59.065078 10837 catalog_manager.cc:4841] TS 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085): tablet cb74c95f705949c8b670b883e327b1dd (table test-table [id=b4e0952753d443458ac97d41bfe7e877]) successfully deleted
I20260925 15:18:59.065826 11404 ts_tablet_manager.cc:1905] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:59.067396 11358 tablet_bootstrap.cc:492] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Bootstrap starting.
I20260925 15:18:59.069322 11358 tablet_bootstrap.cc:654] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:59.070474 11267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3a4002ee3b43558ed8a92e013aa3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:59.072914 11404 ts_tablet_manager.cc:1918] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:18:59.073076 11404 log.cc:1196] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/cb74c95f705949c8b670b883e327b1dd
I20260925 15:18:59.073621 11404 ts_tablet_manager.cc:1939] T cb74c95f705949c8b670b883e327b1dd P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting consensus metadata
I20260925 15:18:59.075196 10837 catalog_manager.cc:4841] TS 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037): tablet cb74c95f705949c8b670b883e327b1dd (table test-table [id=b4e0952753d443458ac97d41bfe7e877]) successfully deleted
I20260925 15:18:59.075908 11359 tablet_bootstrap.cc:492] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: No bootstrap required, opened a new log
I20260925 15:18:59.076082 11359 ts_tablet_manager.cc:1397] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260925 15:18:59.076848 11359 raft_consensus.cc:348] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.077123 11359 raft_consensus.cc:374] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:59.077219 11359 raft_consensus.cc:729] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Initialized, Role: FOLLOWER
I20260925 15:18:59.078297 11358 tablet_bootstrap.cc:492] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: No bootstrap required, opened a new log
I20260925 15:18:59.078454 11358 ts_tablet_manager.cc:1397] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260925 15:18:59.079080 11357 tablet_bootstrap.cc:492] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Bootstrap starting.
I20260925 15:18:59.079211 11358 raft_consensus.cc:348] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.079448 11358 raft_consensus.cc:374] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:59.079526 11358 raft_consensus.cc:729] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4da162339b4049b533914899b5f859, State: Initialized, Role: FOLLOWER
I20260925 15:18:59.079806 11358 consensus_queue.cc:260] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.080765 11357 tablet_bootstrap.cc:654] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:59.081054 11403 ts_tablet_manager.cc:1918] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:18:59.081173 11358 ts_tablet_manager.cc:1428] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:59.081353 11403 log.cc:1196] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/cb74c95f705949c8b670b883e327b1dd
I20260925 15:18:59.081532 11359 consensus_queue.cc:260] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.081945 11403 ts_tablet_manager.cc:1939] T cb74c95f705949c8b670b883e327b1dd P 146888b99cb44933b0a96babf2a5f757: Deleting consensus metadata
I20260925 15:18:59.082522 11359 ts_tablet_manager.cc:1428] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260925 15:18:59.083653 10839 catalog_manager.cc:4841] TS 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689): tablet cb74c95f705949c8b670b883e327b1dd (table test-table [id=b4e0952753d443458ac97d41bfe7e877]) successfully deleted
I20260925 15:18:59.088189 11357 tablet_bootstrap.cc:492] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: No bootstrap required, opened a new log
I20260925 15:18:59.088361 11357 ts_tablet_manager.cc:1397] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260925 15:18:59.089077 11357 raft_consensus.cc:348] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.089290 11357 raft_consensus.cc:374] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:59.089385 11357 raft_consensus.cc:729] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146888b99cb44933b0a96babf2a5f757, State: Initialized, Role: FOLLOWER
I20260925 15:18:59.089651 11357 consensus_queue.cc:260] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.090150 11357 ts_tablet_manager.cc:1428] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:18:59.364503 11363 raft_consensus.cc:482] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:59.364728 11363 raft_consensus.cc:504] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.366443 11363 leader_election.cc:288] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:59.368419 11364 raft_consensus.cc:482] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:59.368661 11364 raft_consensus.cc:504] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.369472 11364 leader_election.cc:288] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:59.370558 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3a4002ee3b43558ed8a92e013aa3e6" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757" is_pre_election: true
I20260925 15:18:59.370882 11293 raft_consensus.cc:2454] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:18:59.371845 10942 leader_election.cc:302] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146888b99cb44933b0a96babf2a5f757, 4b06dc95e81840b6aed3e2b9b1470f9a; no voters:
I20260925 15:18:59.372467 11364 raft_consensus.cc:2781] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:59.372602 11364 raft_consensus.cc:482] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:59.372742 11364 raft_consensus.cc:3037] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:59.376782 11364 raft_consensus.cc:504] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.378571 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3a4002ee3b43558ed8a92e013aa3e6" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757"
I20260925 15:18:59.378810 11293 raft_consensus.cc:3037] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:59.381925 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3a4002ee3b43558ed8a92e013aa3e6" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859" is_pre_election: true
I20260925 15:18:59.382148 11154 raft_consensus.cc:2454] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:18:59.382623 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3a4002ee3b43558ed8a92e013aa3e6" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859"
I20260925 15:18:59.382804 11154 raft_consensus.cc:3037] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:59.383486 11293 raft_consensus.cc:2454] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:18:59.383976 10942 leader_election.cc:302] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146888b99cb44933b0a96babf2a5f757, 4b06dc95e81840b6aed3e2b9b1470f9a; no voters:
I20260925 15:18:59.384279 11399 raft_consensus.cc:2781] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:59.384330 11364 leader_election.cc:288] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Requested vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:18:59.384516 11399 raft_consensus.cc:686] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Becoming Leader. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Running, Role: LEADER
I20260925 15:18:59.384809 11399 consensus_queue.cc:237] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:18:59.385704 11154 raft_consensus.cc:2454] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:18:59.390653 10853 catalog_manager.cc:5495] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65). New cstate: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260925 15:18:59.391013 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:59.366668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:45037 (local address 127.8.226.66:44621)
0925 15:18:59.367343 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:59.367367 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:59.367603 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:59.384817 (+ 17214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:59.384822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:59.384850 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:59.385227 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:59.385240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.387016 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.387019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:59.388297 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:59.388308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.390258 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.390264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:59.390446 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:59.390478 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:18:59.390562 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:18:59.390700 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":886,"threads_started":1}
I20260925 15:18:59.391606 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:59.367085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:44621 (local address 127.8.226.65:45037)
0925 15:18:59.367290 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:59.367296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:59.367465 (+ 169us) server_negotiation.cc:407] Connection header received
0925 15:18:59.367828 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:59.367836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:59.367967 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:59.368042 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:59.385384 (+ 17342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.386489 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.388450 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.389928 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.390100 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:59.390175 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:59.391218 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0925 15:18:59.391358 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:18:59.392642 11010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3a4002ee3b43558ed8a92e013aa3e6" candidate_uuid: "6a4da162339b4049b533914899b5f859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" is_pre_election: true
I20260925 15:18:59.395514 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:59.366726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38689 (local address 127.8.226.66:48413)
0925 15:18:59.386280 (+ 19554us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:59.386299 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:59.386400 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:59.389277 (+ 2877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:59.389282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:59.389305 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:59.389703 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:59.389714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.391150 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.391155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:59.393205 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:59.393220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.394939 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.394948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:59.395103 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:59.395128 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:18:59.395205 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:18:59.395289 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19417,"thread_start_us":14214,"threads_started":1}
I20260925 15:18:59.395684 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:59.367450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:48413 (local address 127.8.226.67:38689)
0925 15:18:59.388792 (+ 21342us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:59.388798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:59.388819 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:18:59.388889 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:59.388896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:59.389013 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:59.389102 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:59.389899 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.390955 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.393416 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:59.394708 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:59.394867 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:59.394972 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:59.395263 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 15:18:59.395407 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21221,"thread_start_us":21100,"threads_started":1}
I20260925 15:18:59.396173 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3a4002ee3b43558ed8a92e013aa3e6" candidate_uuid: "6a4da162339b4049b533914899b5f859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757" is_pre_election: true
I20260925 15:18:59.396493 11293 raft_consensus.cc:2379] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a4da162339b4049b533914899b5f859 in current term 1: Already voted for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in this term.
I20260925 15:18:59.396924 11082 leader_election.cc:302] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4da162339b4049b533914899b5f859; no voters: 146888b99cb44933b0a96babf2a5f757, 4b06dc95e81840b6aed3e2b9b1470f9a
I20260925 15:18:59.397342 11363 raft_consensus.cc:2726] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:18:59.446652 11293 raft_consensus.cc:1261] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:59.447202 11395 consensus_queue.cc:1035] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:59.447800 11154 raft_consensus.cc:1261] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:59.448246 11395 consensus_queue.cc:1035] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:18:59.479418 11383 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:19:01.073771 10851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:01.073953 10851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:01.077464 10851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Sending DeleteTablet for 3 replicas of tablet aa3a4002ee3b43558ed8a92e013aa3e6
I20260925 15:19:01.078099 10989 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3a4002ee3b43558ed8a92e013aa3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260925 15:19:01.078339 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3a4002ee3b43558ed8a92e013aa3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:53092
I20260925 15:19:01.078337 11267 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3a4002ee3b43558ed8a92e013aa3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34426
I20260925 15:19:01.090262 11437 tablet_replica.cc:330] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: stopping tablet replica
I20260925 15:19:01.090564 11437 raft_consensus.cc:2229] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:01.090860 11437 raft_consensus.cc:2258] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:01.091440 11436 tablet_replica.cc:330] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: stopping tablet replica
I20260925 15:19:01.091653 11436 raft_consensus.cc:2229] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:01.091912 11436 raft_consensus.cc:2258] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:01.092056 11437 ts_tablet_manager.cc:1905] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:01.092494 11435 tablet_replica.cc:330] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: stopping tablet replica
I20260925 15:19:01.092705 11435 raft_consensus.cc:2229] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:01.092996 11436 ts_tablet_manager.cc:1905] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:01.093050 11435 raft_consensus.cc:2258] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:01.093914 11435 ts_tablet_manager.cc:1905] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:01.100603 11436 ts_tablet_manager.cc:1918] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:19:01.100802 11436 log.cc:1196] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/aa3a4002ee3b43558ed8a92e013aa3e6
I20260925 15:19:01.100998 11435 ts_tablet_manager.cc:1918] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:19:01.101167 11435 log.cc:1196] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/aa3a4002ee3b43558ed8a92e013aa3e6
I20260925 15:19:01.101330 11436 ts_tablet_manager.cc:1939] T aa3a4002ee3b43558ed8a92e013aa3e6 P 6a4da162339b4049b533914899b5f859: Deleting consensus metadata
I20260925 15:19:01.101579 11435 ts_tablet_manager.cc:1939] T aa3a4002ee3b43558ed8a92e013aa3e6 P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting consensus metadata
I20260925 15:19:01.102512 10837 catalog_manager.cc:4841] TS 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085): tablet aa3a4002ee3b43558ed8a92e013aa3e6 (table test-table [id=bdf54764722a4af1b71b8c77c73565f9]) successfully deleted
I20260925 15:19:01.102773 10837 catalog_manager.cc:4841] TS 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037): tablet aa3a4002ee3b43558ed8a92e013aa3e6 (table test-table [id=bdf54764722a4af1b71b8c77c73565f9]) successfully deleted
I20260925 15:19:01.109351 11437 ts_tablet_manager.cc:1918] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:19:01.109503 11437 log.cc:1196] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/aa3a4002ee3b43558ed8a92e013aa3e6
I20260925 15:19:01.109944 11437 ts_tablet_manager.cc:1939] T aa3a4002ee3b43558ed8a92e013aa3e6 P 146888b99cb44933b0a96babf2a5f757: Deleting consensus metadata
I20260925 15:19:01.110983 10839 catalog_manager.cc:4841] TS 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689): tablet aa3a4002ee3b43558ed8a92e013aa3e6 (table test-table [id=bdf54764722a4af1b71b8c77c73565f9]) successfully deleted
Sep 25 15:19:01 dist-test-slave-8vw7 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:19:01 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:01.133839 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.121054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.8.226.126:44203)
0925 15:19:01.121619 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:01.121623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:01.121643 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:19:01.121796 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:01.121800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:01.121860 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:01.121945 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:01.123142 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.123878 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.125065 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.125513 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.125600 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.129207 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:01.129211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:01.129214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:01.129245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:01.131728 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:01.132271 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:01.132275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:01.132277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:01.132384 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:01.132698 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:01.132701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:01.132703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:01.132946 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:01.133163 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:01.133375 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:19:01.133516 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":310,"threads_started":1}
I20260925 15:19:01.136641 10851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:01.137257 10851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:01.142784 11129 tablet_service.cc:1467] Processing CreateTablet for tablet b4d871324822440da3bbe7c7b0521b62 (DEFAULT_TABLE table=test-table [id=5d9632cde7e547f2af40a1f91f5df635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:01.142777 10989 tablet_service.cc:1467] Processing CreateTablet for tablet b4d871324822440da3bbe7c7b0521b62 (DEFAULT_TABLE table=test-table [id=5d9632cde7e547f2af40a1f91f5df635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:01.143055 11267 tablet_service.cc:1467] Processing CreateTablet for tablet b4d871324822440da3bbe7c7b0521b62 (DEFAULT_TABLE table=test-table [id=5d9632cde7e547f2af40a1f91f5df635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:01.143211 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d871324822440da3bbe7c7b0521b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:01.143226 10989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d871324822440da3bbe7c7b0521b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:01.143478 11267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d871324822440da3bbe7c7b0521b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:01.149411 11446 tablet_bootstrap.cc:492] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Bootstrap starting.
I20260925 15:19:01.150060 11447 tablet_bootstrap.cc:492] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Bootstrap starting.
I20260925 15:19:01.150575 11448 tablet_bootstrap.cc:492] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Bootstrap starting.
I20260925 15:19:01.151187 11446 tablet_bootstrap.cc:654] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:01.152230 11448 tablet_bootstrap.cc:654] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:01.153357 11446 tablet_bootstrap.cc:492] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: No bootstrap required, opened a new log
I20260925 15:19:01.153497 11446 ts_tablet_manager.cc:1397] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:19:01.153985 11447 tablet_bootstrap.cc:654] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:01.154068 11446 raft_consensus.cc:348] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.154219 11448 tablet_bootstrap.cc:492] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: No bootstrap required, opened a new log
I20260925 15:19:01.154263 11446 raft_consensus.cc:374] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:01.154392 11446 raft_consensus.cc:729] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146888b99cb44933b0a96babf2a5f757, State: Initialized, Role: FOLLOWER
I20260925 15:19:01.154403 11448 ts_tablet_manager.cc:1397] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:01.154608 11446 consensus_queue.cc:260] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.155200 11448 raft_consensus.cc:348] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.155563 11448 raft_consensus.cc:374] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:01.155709 11448 raft_consensus.cc:729] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4da162339b4049b533914899b5f859, State: Initialized, Role: FOLLOWER
I20260925 15:19:01.156060 11448 consensus_queue.cc:260] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.157617 11448 ts_tablet_manager.cc:1428] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:01.158517 11446 ts_tablet_manager.cc:1428] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260925 15:19:01.159494 11447 tablet_bootstrap.cc:492] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: No bootstrap required, opened a new log
I20260925 15:19:01.159634 11447 ts_tablet_manager.cc:1397] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 15:19:01.160426 11447 raft_consensus.cc:348] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.160655 11447 raft_consensus.cc:374] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:01.160746 11447 raft_consensus.cc:729] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Initialized, Role: FOLLOWER
I20260925 15:19:01.161017 11447 consensus_queue.cc:260] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.161589 11447 ts_tablet_manager.cc:1428] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:01.370486 11395 raft_consensus.cc:482] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:01.370716 11395 raft_consensus.cc:504] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.371379 11395 leader_election.cc:288] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:19:01.371914 11153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d871324822440da3bbe7c7b0521b62" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859" is_pre_election: true
I20260925 15:19:01.371975 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d871324822440da3bbe7c7b0521b62" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757" is_pre_election: true
I20260925 15:19:01.372140 11153 raft_consensus.cc:2454] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:19:01.372252 11293 raft_consensus.cc:2454] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 0.
I20260925 15:19:01.372620 10940 leader_election.cc:302] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b06dc95e81840b6aed3e2b9b1470f9a, 6a4da162339b4049b533914899b5f859; no voters:
I20260925 15:19:01.372887 11395 raft_consensus.cc:2781] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:01.373037 11395 raft_consensus.cc:482] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:01.373126 11395 raft_consensus.cc:3037] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:01.375029 11395 raft_consensus.cc:504] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.375545 11395 leader_election.cc:288] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Requested vote from peers 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:19:01.375946 11153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d871324822440da3bbe7c7b0521b62" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4da162339b4049b533914899b5f859"
I20260925 15:19:01.376052 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d871324822440da3bbe7c7b0521b62" candidate_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757"
I20260925 15:19:01.376157 11153 raft_consensus.cc:3037] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:01.376255 11293 raft_consensus.cc:3037] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:01.378363 11153 raft_consensus.cc:2454] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:19:01.378451 11293 raft_consensus.cc:2454] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b06dc95e81840b6aed3e2b9b1470f9a in term 1.
I20260925 15:19:01.378755 10940 leader_election.cc:302] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b06dc95e81840b6aed3e2b9b1470f9a, 6a4da162339b4049b533914899b5f859; no voters:
I20260925 15:19:01.379027 11395 raft_consensus.cc:2781] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:01.379172 11395 raft_consensus.cc:686] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Becoming Leader. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Running, Role: LEADER
I20260925 15:19:01.379446 11395 consensus_queue.cc:237] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.381805 10852 catalog_manager.cc:5495] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65). New cstate: current_term: 1 leader_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:01.451524 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.446368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.8.226.65:45037)
0925 15:19:01.446987 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:01.446991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:01.447008 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:01.447054 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:01.447058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:01.447125 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:01.447227 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:01.447620 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.448435 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.449749 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.450210 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.450332 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.450757 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:01.450856 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:01.450948 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:01.451110 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:19:01.451178 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":330,"threads_started":1}
I20260925 15:19:01.461496 11153 raft_consensus.cc:1261] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:01.461591 11293 raft_consensus.cc:1261] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Refusing update from remote peer 4b06dc95e81840b6aed3e2b9b1470f9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:01.462066 11423 consensus_queue.cc:1035] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:01.462424 11430 consensus_queue.cc:1035] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:01.508311 10853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:01.508562 10853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:01.511023 10853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Sending DeleteTablet for 3 replicas of tablet b4d871324822440da3bbe7c7b0521b62
I20260925 15:19:01.511798 10990 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d871324822440da3bbe7c7b0521b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260925 15:19:01.511821 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d871324822440da3bbe7c7b0521b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:53092
I20260925 15:19:01.511855 11267 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d871324822440da3bbe7c7b0521b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34426
I20260925 15:19:01.512065 11436 tablet_replica.cc:330] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: stopping tablet replica
I20260925 15:19:01.512080 11437 tablet_replica.cc:330] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: stopping tablet replica
I20260925 15:19:01.512223 11436 raft_consensus.cc:2229] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:01.512279 11437 raft_consensus.cc:2229] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:01.512404 11436 raft_consensus.cc:2258] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:01.512461 11437 raft_consensus.cc:2258] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:01.512703 11435 tablet_replica.cc:330] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: stopping tablet replica
I20260925 15:19:01.512892 11435 raft_consensus.cc:2229] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:01.512778 10853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:19:01.513206 11435 raft_consensus.cc:2258] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 15:19:01.513382 10853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:01.513535 11436 ts_tablet_manager.cc:1905] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:01.514232 11437 ts_tablet_manager.cc:1905] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:01.514264 11435 ts_tablet_manager.cc:1905] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:01.519851 11129 tablet_service.cc:1467] Processing CreateTablet for tablet 37a2eb0af67c487b8e47d19c36e9d93e (DEFAULT_TABLE table=test-table [id=27d5d1c736424c27bc34443832f98dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:01.520020 11267 tablet_service.cc:1467] Processing CreateTablet for tablet 37a2eb0af67c487b8e47d19c36e9d93e (DEFAULT_TABLE table=test-table [id=27d5d1c736424c27bc34443832f98dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:01.520265 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a2eb0af67c487b8e47d19c36e9d93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:01.520402 11267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a2eb0af67c487b8e47d19c36e9d93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:01.520732 11437 ts_tablet_manager.cc:1918] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:01.520874 11437 log.cc:1196] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/b4d871324822440da3bbe7c7b0521b62
I20260925 15:19:01.521317 11437 ts_tablet_manager.cc:1939] T b4d871324822440da3bbe7c7b0521b62 P 146888b99cb44933b0a96babf2a5f757: Deleting consensus metadata
I20260925 15:19:01.522523 10839 catalog_manager.cc:4841] TS 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689): tablet b4d871324822440da3bbe7c7b0521b62 (table test-table [id=5d9632cde7e547f2af40a1f91f5df635]) successfully deleted
I20260925 15:19:01.524076 11435 ts_tablet_manager.cc:1918] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:01.524225 11435 log.cc:1196] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/b4d871324822440da3bbe7c7b0521b62
I20260925 15:19:01.524607 11435 ts_tablet_manager.cc:1939] T b4d871324822440da3bbe7c7b0521b62 P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting consensus metadata
I20260925 15:19:01.525053 10990 tablet_service.cc:1467] Processing CreateTablet for tablet 37a2eb0af67c487b8e47d19c36e9d93e (DEFAULT_TABLE table=test-table [id=27d5d1c736424c27bc34443832f98dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:01.525426 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a2eb0af67c487b8e47d19c36e9d93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:01.525840 11448 tablet_bootstrap.cc:492] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Bootstrap starting.
I20260925 15:19:01.526652 11436 ts_tablet_manager.cc:1918] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:01.526813 11436 log.cc:1196] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/b4d871324822440da3bbe7c7b0521b62
I20260925 15:19:01.527202 11436 ts_tablet_manager.cc:1939] T b4d871324822440da3bbe7c7b0521b62 P 6a4da162339b4049b533914899b5f859: Deleting consensus metadata
I20260925 15:19:01.527395 11448 tablet_bootstrap.cc:654] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:01.527745 11446 tablet_bootstrap.cc:492] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Bootstrap starting.
I20260925 15:19:01.529201 10837 catalog_manager.cc:4841] TS 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085): tablet b4d871324822440da3bbe7c7b0521b62 (table test-table [id=5d9632cde7e547f2af40a1f91f5df635]) successfully deleted
I20260925 15:19:01.529441 11446 tablet_bootstrap.cc:654] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:01.529419 10837 catalog_manager.cc:4841] TS 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037): tablet b4d871324822440da3bbe7c7b0521b62 (table test-table [id=5d9632cde7e547f2af40a1f91f5df635]) successfully deleted
I20260925 15:19:01.530669 11448 tablet_bootstrap.cc:492] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: No bootstrap required, opened a new log
I20260925 15:19:01.530804 11448 ts_tablet_manager.cc:1397] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 15:19:01.531555 11447 tablet_bootstrap.cc:492] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Bootstrap starting.
I20260925 15:19:01.531620 11448 raft_consensus.cc:348] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.531883 11448 raft_consensus.cc:374] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:01.531980 11448 raft_consensus.cc:729] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4da162339b4049b533914899b5f859, State: Initialized, Role: FOLLOWER
I20260925 15:19:01.532003 11446 tablet_bootstrap.cc:492] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: No bootstrap required, opened a new log
I20260925 15:19:01.532140 11446 ts_tablet_manager.cc:1397] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:01.532267 11448 consensus_queue.cc:260] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.532814 11448 ts_tablet_manager.cc:1428] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:19:01.532842 11446 raft_consensus.cc:348] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.533111 11447 tablet_bootstrap.cc:654] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:01.533414 11446 raft_consensus.cc:374] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:01.533519 11446 raft_consensus.cc:729] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146888b99cb44933b0a96babf2a5f757, State: Initialized, Role: FOLLOWER
I20260925 15:19:01.533758 11446 consensus_queue.cc:260] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.534319 11446 ts_tablet_manager.cc:1428] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:01.535071 11447 tablet_bootstrap.cc:492] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: No bootstrap required, opened a new log
I20260925 15:19:01.535239 11447 ts_tablet_manager.cc:1397] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:19:01.535844 11447 raft_consensus.cc:348] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.536082 11447 raft_consensus.cc:374] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:01.536158 11447 raft_consensus.cc:729] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06dc95e81840b6aed3e2b9b1470f9a, State: Initialized, Role: FOLLOWER
I20260925 15:19:01.536391 11447 consensus_queue.cc:260] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.536830 11447 ts_tablet_manager.cc:1428] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:19:01.729238 11453 raft_consensus.cc:482] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:01.729449 11453 raft_consensus.cc:504] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.730122 11453 leader_election.cc:288] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:19:01.730679 11010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a2eb0af67c487b8e47d19c36e9d93e" candidate_uuid: "6a4da162339b4049b533914899b5f859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" is_pre_election: true
I20260925 15:19:01.730739 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a2eb0af67c487b8e47d19c36e9d93e" candidate_uuid: "6a4da162339b4049b533914899b5f859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757" is_pre_election: true
I20260925 15:19:01.730973 11293 raft_consensus.cc:2454] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4da162339b4049b533914899b5f859 in term 0.
I20260925 15:19:01.731022 11010 raft_consensus.cc:2454] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4da162339b4049b533914899b5f859 in term 0.
I20260925 15:19:01.731400 11082 leader_election.cc:302] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146888b99cb44933b0a96babf2a5f757, 6a4da162339b4049b533914899b5f859; no voters:
I20260925 15:19:01.731662 11453 raft_consensus.cc:2781] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:01.731792 11453 raft_consensus.cc:482] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:01.731884 11453 raft_consensus.cc:3037] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:01.733960 11453 raft_consensus.cc:504] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.734505 11453 leader_election.cc:288] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [CANDIDATE]: Term 1 election: Requested vote from peers 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037), 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689)
I20260925 15:19:01.734894 11010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a2eb0af67c487b8e47d19c36e9d93e" candidate_uuid: "6a4da162339b4049b533914899b5f859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a"
I20260925 15:19:01.734992 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a2eb0af67c487b8e47d19c36e9d93e" candidate_uuid: "6a4da162339b4049b533914899b5f859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146888b99cb44933b0a96babf2a5f757"
I20260925 15:19:01.735111 11010 raft_consensus.cc:3037] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:01.735198 11293 raft_consensus.cc:3037] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:01.738116 11010 raft_consensus.cc:2454] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4da162339b4049b533914899b5f859 in term 1.
I20260925 15:19:01.738132 11293 raft_consensus.cc:2454] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4da162339b4049b533914899b5f859 in term 1.
I20260925 15:19:01.738511 11080 leader_election.cc:302] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b06dc95e81840b6aed3e2b9b1470f9a, 6a4da162339b4049b533914899b5f859; no voters:
I20260925 15:19:01.738808 11453 raft_consensus.cc:2781] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:01.739212 11453 raft_consensus.cc:686] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 1 LEADER]: Becoming Leader. State: Replica: 6a4da162339b4049b533914899b5f859, State: Running, Role: LEADER
I20260925 15:19:01.739589 11453 consensus_queue.cc:237] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } }
I20260925 15:19:01.742604 10852 catalog_manager.cc:5495] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a4da162339b4049b533914899b5f859 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "6a4da162339b4049b533914899b5f859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4da162339b4049b533914899b5f859" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:01.777452 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.772207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.8.226.66:36085)
0925 15:19:01.772809 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:01.772813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:01.772831 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:19:01.772906 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:01.772909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:01.773032 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:01.773166 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:01.773502 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.774253 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.775449 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.775825 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.775906 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.776241 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:01.776820 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:01.776927 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:01.777161 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:19:01.777221 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":319,"threads_started":1}
I20260925 15:19:01.790544 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.786023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:50453 (local address 127.8.226.65:45037)
0925 15:19:01.786161 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:01.786165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:01.786629 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:19:01.786749 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:01.786753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:01.786812 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:01.786899 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:01.787299 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.787960 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.789207 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.789617 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.789703 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.790053 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:01.790144 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:01.790203 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:01.790312 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:19:01.790363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:19:01.790705 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.786036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:45037 (local address 127.8.226.66:50453)
0925 15:19:01.786540 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:01.786555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:01.786654 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:01.786924 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:01.786927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:01.786936 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:01.787150 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:01.787156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.788059 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.788062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:01.789108 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:01.789119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.789750 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.789755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:01.789865 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.789911 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:01.790223 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:01.790230 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:19:01.790284 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:19:01.790368 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
I20260925 15:19:01.797591 11293 raft_consensus.cc:1261] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Refusing update from remote peer 6a4da162339b4049b533914899b5f859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:01.797874 11010 raft_consensus.cc:1261] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Refusing update from remote peer 6a4da162339b4049b533914899b5f859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:01.798164 11463 consensus_queue.cc:1035] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146888b99cb44933b0a96babf2a5f757" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:01.798535 11453 consensus_queue.cc:1035] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b06dc95e81840b6aed3e2b9b1470f9a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 45037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:01.820613 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.815468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:42891 (local address 127.8.226.66:36085)
0925 15:19:01.815608 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:01.815612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:01.815986 (+ 374us) server_negotiation.cc:407] Connection header received
0925 15:19:01.816179 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:01.816183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:01.816259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:01.816350 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:01.816798 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.817546 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.819280 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.819768 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.819850 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.820164 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:01.820245 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:01.820303 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:01.820395 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:19:01.820443 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:19:01.820698 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:01.815354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36085 (local address 127.8.226.65:42891)
0925 15:19:01.815896 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:01.815910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:01.816036 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:01.816379 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:01.816385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:01.816397 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:01.816676 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:01.816685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.817649 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.817653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:01.819134 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:01.819161 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:01.819909 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:01.819914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:01.820015 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:01.820046 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:01.820325 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:01.820332 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:19:01.820384 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:19:01.820467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":250,"threads_started":1}
W20260925 15:19:01.821519 11419 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:19:03.394246 10851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:03.394503 10851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:03.398121 10851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4655c270c254d20bb54a90e339e2509: Sending DeleteTablet for 3 replicas of tablet 37a2eb0af67c487b8e47d19c36e9d93e
I20260925 15:19:03.398766 10989 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a2eb0af67c487b8e47d19c36e9d93e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:03 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260925 15:19:03.398833 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a2eb0af67c487b8e47d19c36e9d93e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:03 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:53092
I20260925 15:19:03.398938 11267 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a2eb0af67c487b8e47d19c36e9d93e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:03 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34426
I20260925 15:19:03.400068 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10924
I20260925 15:19:03.411595 11499 tablet_replica.cc:330] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: stopping tablet replica
I20260925 15:19:03.411792 11501 tablet_replica.cc:330] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: stopping tablet replica
I20260925 15:19:03.412206 11500 tablet_replica.cc:330] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: stopping tablet replica
I20260925 15:19:03.412248 11499 raft_consensus.cc:2229] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:03.412442 11500 raft_consensus.cc:2229] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:03.412639 11499 raft_consensus.cc:2258] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:03.412935 11500 raft_consensus.cc:2258] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:03.413259 11501 raft_consensus.cc:2229] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:03.413540 11501 raft_consensus.cc:2258] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:03.413986 11499 ts_tablet_manager.cc:1905] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:03.414597 11500 ts_tablet_manager.cc:1905] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:03.415273 11501 ts_tablet_manager.cc:1905] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:03.420507 11499 ts_tablet_manager.cc:1918] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:19:03.420727 11499 log.cc:1196] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/37a2eb0af67c487b8e47d19c36e9d93e
I20260925 15:19:03.420864 11500 ts_tablet_manager.cc:1918] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:19:03.421085 11500 log.cc:1196] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/37a2eb0af67c487b8e47d19c36e9d93e
I20260925 15:19:03.421459 11499 ts_tablet_manager.cc:1939] T 37a2eb0af67c487b8e47d19c36e9d93e P 4b06dc95e81840b6aed3e2b9b1470f9a: Deleting consensus metadata
I20260925 15:19:03.421588 11500 ts_tablet_manager.cc:1939] T 37a2eb0af67c487b8e47d19c36e9d93e P 6a4da162339b4049b533914899b5f859: Deleting consensus metadata
I20260925 15:19:03.422617 10837 catalog_manager.cc:4841] TS 4b06dc95e81840b6aed3e2b9b1470f9a (127.8.226.65:45037): tablet 37a2eb0af67c487b8e47d19c36e9d93e (table test-table [id=27d5d1c736424c27bc34443832f98dd2]) successfully deleted
I20260925 15:19:03.422899 10837 catalog_manager.cc:4841] TS 6a4da162339b4049b533914899b5f859 (127.8.226.66:36085): tablet 37a2eb0af67c487b8e47d19c36e9d93e (table test-table [id=27d5d1c736424c27bc34443832f98dd2]) successfully deleted
I20260925 15:19:03.425427 11501 ts_tablet_manager.cc:1918] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:19:03.425567 11501 log.cc:1196] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/37a2eb0af67c487b8e47d19c36e9d93e
I20260925 15:19:03.425992 11501 ts_tablet_manager.cc:1939] T 37a2eb0af67c487b8e47d19c36e9d93e P 146888b99cb44933b0a96babf2a5f757: Deleting consensus metadata
I20260925 15:19:03.426900 10839 catalog_manager.cc:4841] TS 146888b99cb44933b0a96babf2a5f757 (127.8.226.67:38689): tablet 37a2eb0af67c487b8e47d19c36e9d93e (table test-table [id=27d5d1c736424c27bc34443832f98dd2]) successfully deleted
Sep 25 15:19:03 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:03.436877 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:03.401191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.8.226.65:45037)
0925 15:19:03.423067 (+ 21876us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:03.423072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:03.423468 (+ 396us) server_negotiation.cc:407] Connection header received
0925 15:19:03.423600 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:03.423606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:03.423686 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:03.423802 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:03.425079 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:03.426243 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:03.427634 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:03.428005 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:03.428112 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:03.431831 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:03.431845 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:03.431860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:03.431891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:03.434535 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:03.435032 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:03.435038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:03.435040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:03.435158 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:03.435526 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:03.435530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:03.435532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:03.435878 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:03.436079 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:03.436304 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:19:03.436529 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21746,"thread_start_us":21606,"threads_started":1}
W20260925 15:19:03.437628 11055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45504
W20260925 15:19:03.438207 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10924: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:19:03.438413 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 10924
I20260925 15:19:03.462972 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11064
Sep 25 15:19:03 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:19:03.478633 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:03.463621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.8.226.66:36085)
0925 15:19:03.464107 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:03.464112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:03.464127 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:19:03.464177 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:03.464180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:03.464241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:03.464332 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:03.465489 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:03.466300 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:03.467580 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:03.467947 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:03.468030 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:03.471976 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:03.471988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:03.472002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:03.472049 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:03.475619 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:03.476244 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:03.476252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:03.476256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:03.476385 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:03.476727 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:03.476744 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:03.476747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:03.477511 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:03.477810 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:03.478069 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:19:03.478288 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":219,"threads_started":1}
W20260925 15:19:03.479278 11194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122
W20260925 15:19:03.479954 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:19:03.480068 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 11064
I20260925 15:19:03.501163 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11203
Sep 25 15:19:03 dist-test-slave-8vw7 krb5kdc[10808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:19:03.520153 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:03.501851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.8.226.67:38689)
0925 15:19:03.502397 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:03.502414 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:03.502428 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:19:03.502477 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:03.502479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:03.502538 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:03.502640 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:03.503802 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:03.504538 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:03.505850 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:03.506227 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:03.506322 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:03.515075 (+ 8753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:03.515089 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:03.515104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:03.515137 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:03.517939 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:03.518503 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:03.518509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:03.518512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:03.518633 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:03.518926 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:03.518929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:03.518931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:03.519252 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:03.519461 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:03.519648 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 15:19:03.519800 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":289,"threads_started":1}
W20260925 15:19:03.520787 11333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34428
W20260925 15:19:03.521452 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:19:03.521624 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 11203
I20260925 15:19:03.539129 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 10821
I20260925 15:19:03.540151 10893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:03.652486 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 10821
2026-09-25T15:19:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8423 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:03 dist-test-slave-8vw7 krb5kdc[11509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:03 dist-test-slave-8vw7 krb5kdc[11509](info): set up 2 sockets
Sep 25 15:19:03 dist-test-slave-8vw7 krb5kdc[11509](info): commencing operation
krb5kdc: starting...
W20260925 15:19:04.693107 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:04 dist-test-slave-8vw7 krb5kdc[11509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:04.861814 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44375
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:44375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:05.165409 11522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:05.165683 11522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:05.165829 11522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:05.172796 11522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:05.172910 11522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:05.173015 11522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:05.173095 11522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:05.173193 11522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:05.182401 11522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:44375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:05.183600 11522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:05.184855 11522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:05.196190 11530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:05.196223 11528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:05.196424 11527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:05.196321 11522 server_base.cc:1034] running on GCE node
I20260925 15:19:05.197463 11522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:05.198683 11522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:05.199887 11522 hybrid_clock.cc:648] HybridClock initialized: now 1790349545199839 us; error 29 us; skew 500 ppm
I20260925 15:19:05.200220 11522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:05 dist-test-slave-8vw7 krb5kdc[11509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:05.203187 11522 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:05.204564 11522 webserver.cc:460] Webserver started at http://127.8.226.126:43641/ using document root <none> and password file <none>
I20260925 15:19:05.205291 11522 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:05.205427 11522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:05.205757 11522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:05.208487 11522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "61f6c4e867e74eb68e5a2fefc423b84c"
format_stamp: "Formatted at 2026-09-25 15:19:05 on dist-test-slave-8vw7"
server_key: "9b19c6f91c62eab93a806e14b6f74d15"
server_key_iv: "23cfbb6fe830f9614b609d4adf57ca84"
server_key_version: "encryptionkey@0"
I20260925 15:19:05.209260 11522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "61f6c4e867e74eb68e5a2fefc423b84c"
format_stamp: "Formatted at 2026-09-25 15:19:05 on dist-test-slave-8vw7"
server_key: "9b19c6f91c62eab93a806e14b6f74d15"
server_key_iv: "23cfbb6fe830f9614b609d4adf57ca84"
server_key_version: "encryptionkey@0"
I20260925 15:19:05.214222 11522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:19:05.217453 11537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:05.218685 11522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:19:05.218859 11522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "61f6c4e867e74eb68e5a2fefc423b84c"
format_stamp: "Formatted at 2026-09-25 15:19:05 on dist-test-slave-8vw7"
server_key: "9b19c6f91c62eab93a806e14b6f74d15"
server_key_iv: "23cfbb6fe830f9614b609d4adf57ca84"
server_key_version: "encryptionkey@0"
I20260925 15:19:05.219035 11522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:05.234817 11522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:05.238848 11522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:05.239111 11522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:05.256547 11522 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:44375
I20260925 15:19:05.256554 11588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:44375 every 8 connection(s)
I20260925 15:19:05.257913 11522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:05.261319 11589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:05.266724 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 11522
I20260925 15:19:05.266963 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:05.267719 9097 external_mini_cluster.cc:1428] Setting key b133ecd33648c09310aa443e9cdd673f
I20260925 15:19:05.271685 11589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c: Bootstrap starting.
I20260925 15:19:05.275120 11589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:05.276072 11589 log.cc:824] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:05.278963 11589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c: No bootstrap required, opened a new log
Sep 25 15:19:05 dist-test-slave-8vw7 krb5kdc[11509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:05.287459 11589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44375 } }
I20260925 15:19:05.287889 11589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:05.288003 11589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f6c4e867e74eb68e5a2fefc423b84c, State: Initialized, Role: FOLLOWER
I20260925 15:19:05.288617 11589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44375 } }
I20260925 15:19:05.288852 11589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:05.289017 11589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:05.289211 11589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:05.289553 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:05.269622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.8.226.126:44375)
0925 15:19:05.270532 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:05.270543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:05.270582 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:19:05.272267 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:05.272287 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:05.272624 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:05.272942 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:05.274700 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:05.275983 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:05.277648 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:05.278262 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:05.278379 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:05.282674 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:05.282683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:05.282696 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:05.282733 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:05.285556 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:05.286285 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:05.286293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:05.286315 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:05.286462 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:05.286833 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:05.286836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:05.286838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:05.287315 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:05.287522 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:05.288013 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 15:19:05.288435 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":408,"threads_started":1}
I20260925 15:19:05.291438 11589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44375 } }
I20260925 15:19:05.291874 11589 leader_election.cc:302] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f6c4e867e74eb68e5a2fefc423b84c; no voters:
I20260925 15:19:05.292380 11589 leader_election.cc:288] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:05.292481 11594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:05.293171 11594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [term 1 LEADER]: Becoming Leader. State: Replica: 61f6c4e867e74eb68e5a2fefc423b84c, State: Running, Role: LEADER
I20260925 15:19:05.293646 11594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44375 } }
I20260925 15:19:05.294203 11589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:05.296211 11595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44375 } } }
I20260925 15:19:05.296186 11596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f6c4e867e74eb68e5a2fefc423b84c. Latest consensus state: current_term: 1 leader_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f6c4e867e74eb68e5a2fefc423b84c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44375 } } }
I20260925 15:19:05.296574 11595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:05.296640 11596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:05.298538 11599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:05.304587 11599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:05.313324 11599 catalog_manager.cc:1347] Generated new cluster ID: 24b1d2f46da7414fb53aa6a205d6756c
I20260925 15:19:05.313428 11599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:05.329412 11599 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:05.330817 11599 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:05.338961 11599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f6c4e867e74eb68e5a2fefc423b84c: Generated new TSK 0
I20260925 15:19:05.339664 11599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:19:05.390938 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44375
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:05.711427 11615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:05.711697 11615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:05.711802 11615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:05.718677 11615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:05.718798 11615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:05.718902 11615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:05.728534 11615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:05.729753 11615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:05.730952 11615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:05.743074 11623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:05.743033 11621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:05.743111 11620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:05.743884 11615 server_base.cc:1034] running on GCE node
I20260925 15:19:05.744453 11615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:05.745363 11615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:05.746564 11615 hybrid_clock.cc:648] HybridClock initialized: now 1790349545746516 us; error 41 us; skew 500 ppm
I20260925 15:19:05.746850 11615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:05 dist-test-slave-8vw7 krb5kdc[11509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:05.749887 11615 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:19:05.751171 11615 webserver.cc:460] Webserver started at http://127.8.226.65:46195/ using document root <none> and password file <none>
I20260925 15:19:05.751816 11615 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:05.751943 11615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:05.752225 11615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:05.754958 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "279b0fb33db34b43a7d2f05b785d08bf"
format_stamp: "Formatted at 2026-09-25 15:19:05 on dist-test-slave-8vw7"
server_key: "d6f99f0ea913c05f49bba0e1680737fa"
server_key_iv: "819ba48c4e02680aa04f34e9f0243ebf"
server_key_version: "encryptionkey@0"
I20260925 15:19:05.755685 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "279b0fb33db34b43a7d2f05b785d08bf"
format_stamp: "Formatted at 2026-09-25 15:19:05 on dist-test-slave-8vw7"
server_key: "d6f99f0ea913c05f49bba0e1680737fa"
server_key_iv: "819ba48c4e02680aa04f34e9f0243ebf"
server_key_version: "encryptionkey@0"
I20260925 15:19:05.760265 11615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:19:05.763428 11630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:05.764551 11615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:05.764699 11615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "279b0fb33db34b43a7d2f05b785d08bf"
format_stamp: "Formatted at 2026-09-25 15:19:05 on dist-test-slave-8vw7"
server_key: "d6f99f0ea913c05f49bba0e1680737fa"
server_key_iv: "819ba48c4e02680aa04f34e9f0243ebf"
server_key_version: "encryptionkey@0"
I20260925 15:19:05.764894 11615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:05.775458 11615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:05.778996 11615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:05.779245 11615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:05.780035 11615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:05.781230 11615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:05.781329 11615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:05.781452 11615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:05.781515 11615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:05.813416 11615 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:44869
I20260925 15:19:05.813455 11742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:44869 every 8 connection(s)
I20260925 15:19:05.814667 11615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:05.819439 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 11615
I20260925 15:19:05.819657 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:05.820225 9097 external_mini_cluster.cc:1428] Setting key fcd3b5248339ea7563918acb422d1dd0
Sep 25 15:19:05 dist-test-slave-8vw7 krb5kdc[11509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:05.835378 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:05.817072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:37631 (local address 127.8.226.126:44375)
0925 15:19:05.817506 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:05.817510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:05.818542 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 15:19:05.820436 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:05.820441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:05.820520 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:05.820638 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:05.822346 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:05.823233 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:05.824794 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:05.825268 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:05.825395 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:05.828975 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:05.828983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:05.828991 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:05.829032 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:05.832401 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:05.833094 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:05.833102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:05.833107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:05.833244 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:05.833610 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:05.833617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:05.833630 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:05.833988 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:05.834125 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:05.834729 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 15:19:05.834969 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260925 15:19:05.836167 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:05.817685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44375 (local address 127.8.226.65:37631)
0925 15:19:05.818385 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:05.818423 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:05.820132 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:05.820818 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:05.820837 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:05.821460 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:05.822203 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:05.822221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:05.823379 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:05.823384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:05.824661 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:05.824672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:05.825488 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:05.825498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:05.825739 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:05.826477 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:05.826508 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:05.828762 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:05.832521 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:05.832527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:05.832540 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:05.832931 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:05.833341 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:05.833344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:05.833346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:05.833504 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:05.834100 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:05.834105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:05.834385 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 15:19:05.834583 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 15:19:05.835019 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":272,"threads_started":1}
I20260925 15:19:05.837739 11743 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44375
I20260925 15:19:05.838070 11743 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:05.838982 11743 heartbeater.cc:511] Master 127.8.226.126:44375 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:19:05.841146 11554 ts_manager.cc:194] Registered new tserver with Master: 279b0fb33db34b43a7d2f05b785d08bf (127.8.226.65:44869)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
I20260925 15:19:05.843163 11554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:37631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:19:05.862270 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44375
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:06.168135 11749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:06.168490 11749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:06.168654 11749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:06.175575 11749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:06.175668 11749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:06.175812 11749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:06.184871 11749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:06.186136 11749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:06.187400 11749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:06.200089 11757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:06.200136 11755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:06.200050 11754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:06.206281 11749 server_base.cc:1034] running on GCE node
I20260925 15:19:06.206884 11749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:06.207808 11749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:06.209039 11749 hybrid_clock.cc:648] HybridClock initialized: now 1790349546208979 us; error 51 us; skew 500 ppm
I20260925 15:19:06.209348 11749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:06 dist-test-slave-8vw7 krb5kdc[11509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:06.212257 11749 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:19:06.213775 11749 webserver.cc:460] Webserver started at http://127.8.226.66:42309/ using document root <none> and password file <none>
I20260925 15:19:06.214429 11749 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:06.214524 11749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:06.214783 11749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:06.217553 11749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "0511c3b7c21f4fda875e6a6e16f0660c"
format_stamp: "Formatted at 2026-09-25 15:19:06 on dist-test-slave-8vw7"
server_key: "82ecac9db563c32936a339e63b7879ec"
server_key_iv: "fae157f0d77ec30cadeba9368fa5e2e2"
server_key_version: "encryptionkey@0"
I20260925 15:19:06.218266 11749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "0511c3b7c21f4fda875e6a6e16f0660c"
format_stamp: "Formatted at 2026-09-25 15:19:06 on dist-test-slave-8vw7"
server_key: "82ecac9db563c32936a339e63b7879ec"
server_key_iv: "fae157f0d77ec30cadeba9368fa5e2e2"
server_key_version: "encryptionkey@0"
I20260925 15:19:06.223212 11749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:19:06.226459 11764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:06.227792 11749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:19:06.227969 11749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "0511c3b7c21f4fda875e6a6e16f0660c"
format_stamp: "Formatted at 2026-09-25 15:19:06 on dist-test-slave-8vw7"
server_key: "82ecac9db563c32936a339e63b7879ec"
server_key_iv: "fae157f0d77ec30cadeba9368fa5e2e2"
server_key_version: "encryptionkey@0"
I20260925 15:19:06.228174 11749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:06.249116 11749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:06.252808 11749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:06.253158 11749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:06.254021 11749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:06.255201 11749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:06.255285 11749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:06.255403 11749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:06.255461 11749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:06.286412 11749 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:41479
I20260925 15:19:06.286463 11876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:41479 every 8 connection(s)
I20260925 15:19:06.287626 11749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:06.288450 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 11749
I20260925 15:19:06.288638 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:06.289191 9097 external_mini_cluster.cc:1428] Setting key a8c686b79f49e9031c8913cc115253c6
Sep 25 15:19:06 dist-test-slave-8vw7 krb5kdc[11509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:19:06.310518 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:06.290781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:58735 (local address 127.8.226.126:44375)
0925 15:19:06.290971 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:06.290976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:06.292248 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 15:19:06.294957 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:06.294964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:06.295042 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:06.295149 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:06.297014 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.298248 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.299825 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.300550 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.300672 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:06.304033 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:06.304040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:06.304046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:06.304092 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:06.307696 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:06.308405 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:06.308410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:06.308414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:06.308569 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:06.309019 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:06.309024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:06.309027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:06.309284 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:06.309392 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:06.310113 (+ 721us) server_negotiation.cc:299] Negotiation successful
0925 15:19:06.310302 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
I20260925 15:19:06.311517 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:06.291296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44375 (local address 127.8.226.66:58735)
0925 15:19:06.292077 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:06.292123 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:06.294703 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:06.295370 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:06.295382 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:06.296072 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:06.296822 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:06.296849 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.298452 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.298456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:06.299682 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:06.299694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.300769 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.300776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:06.301028 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:06.301664 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:06.301685 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:06.303795 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:06.307847 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:06.307855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:06.307871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:06.308287 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:06.308680 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:06.308683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:06.308701 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:06.308889 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:06.309383 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:06.309388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:06.309690 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 15:19:06.310039 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 15:19:06.310415 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":279,"threads_started":1}
I20260925 15:19:06.313004 11877 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44375
I20260925 15:19:06.313318 11877 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:06.313944 11877 heartbeater.cc:511] Master 127.8.226.126:44375 requested a full tablet report, sending...
I20260925 15:19:06.315292 11554 ts_manager.cc:194] Registered new tserver with Master: 0511c3b7c21f4fda875e6a6e16f0660c (127.8.226.66:41479)
I20260925 15:19:06.316064 11554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:58735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:19:06.333498 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44375
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:06.639282 11882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:06.639668 11882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:06.639850 11882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:06.646981 11882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:06.647095 11882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:06.647205 11882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:06.656468 11882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:06.657752 11882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:06.659053 11882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:06.671578 11890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:06.671672 11888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:06.671662 11887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:06.672322 11882 server_base.cc:1034] running on GCE node
I20260925 15:19:06.672927 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:06.673878 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:06.675094 11882 hybrid_clock.cc:648] HybridClock initialized: now 1790349546675058 us; error 44 us; skew 500 ppm
I20260925 15:19:06.675395 11882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:06 dist-test-slave-8vw7 krb5kdc[11509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:06.678390 11882 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:19:06.679833 11882 webserver.cc:460] Webserver started at http://127.8.226.67:33175/ using document root <none> and password file <none>
I20260925 15:19:06.680495 11882 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:06.680640 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:06.680940 11882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:06.683825 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "7f810a154d404238abfb39472e6f2e9f"
format_stamp: "Formatted at 2026-09-25 15:19:06 on dist-test-slave-8vw7"
server_key: "481402ba3a4d4b9087b2c5fa7ffedd1b"
server_key_iv: "9b76e1602a0fbce93b9f0f7a593f7bc5"
server_key_version: "encryptionkey@0"
I20260925 15:19:06.684562 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "7f810a154d404238abfb39472e6f2e9f"
format_stamp: "Formatted at 2026-09-25 15:19:06 on dist-test-slave-8vw7"
server_key: "481402ba3a4d4b9087b2c5fa7ffedd1b"
server_key_iv: "9b76e1602a0fbce93b9f0f7a593f7bc5"
server_key_version: "encryptionkey@0"
I20260925 15:19:06.689448 11882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 15:19:06.692766 11897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:06.694041 11882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:06.694211 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "7f810a154d404238abfb39472e6f2e9f"
format_stamp: "Formatted at 2026-09-25 15:19:06 on dist-test-slave-8vw7"
server_key: "481402ba3a4d4b9087b2c5fa7ffedd1b"
server_key_iv: "9b76e1602a0fbce93b9f0f7a593f7bc5"
server_key_version: "encryptionkey@0"
I20260925 15:19:06.694417 11882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:06.718715 11882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:06.722601 11882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:06.722920 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:06.723902 11882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:06.725165 11882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:06.725260 11882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:06.725414 11882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:06.725481 11882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:06.757004 11882 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:39461
I20260925 15:19:06.757066 12009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:39461 every 8 connection(s)
I20260925 15:19:06.758327 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:06.760596 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 11882
I20260925 15:19:06.760802 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:06.761307 9097 external_mini_cluster.cc:1428] Setting key 623e2890106761baad98efd055d4f731
Sep 25 15:19:06 dist-test-slave-8vw7 krb5kdc[11509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:06.777928 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:06.760843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:59743 (local address 127.8.226.126:44375)
0925 15:19:06.761052 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:06.761060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:06.762404 (+ 1344us) server_negotiation.cc:407] Connection header received
0925 15:19:06.764713 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:06.764718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:06.764814 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:06.764925 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:06.766372 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.767092 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.768758 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.769253 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.769349 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:06.772525 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:06.772530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:06.772533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:06.772563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:06.774971 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:06.775814 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:06.775818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:06.775820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:06.775958 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:06.776392 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:06.776396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:06.776399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:06.776680 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:06.776793 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:06.777546 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 15:19:06.777693 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:19:06.778820 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:06.761451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44375 (local address 127.8.226.67:59743)
0925 15:19:06.762228 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:06.762288 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:06.764473 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:06.765182 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:06.765190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:06.765676 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:06.766210 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:06.766225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.767233 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.767246 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:06.768559 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:06.768570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:06.769424 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:06.769429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:06.769627 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:06.770266 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:06.770287 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:06.772343 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:06.775089 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:06.775097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:06.775112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:06.775691 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:06.776058 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:06.776061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:06.776063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:06.776289 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:06.776793 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:06.776798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:06.777167 (+ 369us) client_negotiation.cc:769] Sending connection context
0925 15:19:06.777475 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 15:19:06.777820 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
I20260925 15:19:06.779989 12010 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44375
I20260925 15:19:06.780254 12010 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:06.780916 12010 heartbeater.cc:511] Master 127.8.226.126:44375 requested a full tablet report, sending...
I20260925 15:19:06.782158 11554 ts_manager.cc:194] Registered new tserver with Master: 7f810a154d404238abfb39472e6f2e9f (127.8.226.67:39461)
I20260925 15:19:06.782853 11554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:59743
I20260925 15:19:06.788170 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:19:06.816911 11745 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:06.814300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.8.226.126:44375)
0925 15:19:06.814499 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:06.814507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:06.815050 (+ 543us) server_negotiation.cc:407] Connection header received
0925 15:19:06.815217 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:06.815221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:06.815286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:06.815367 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:06.816659 (+ 1292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32990: BlockingRecv error: recv got EOF from 127.0.0.1:32990 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:19:06.820570 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11615
W20260925 15:19:06.823173 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11615: Not authorized: Client connection negotiation failed: client connection to 127.8.226.65:44869: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:19:06.823352 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 11615
I20260925 15:19:06.838094 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11749
W20260925 15:19:06.840476 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11749: Not authorized: Client connection negotiation failed: client connection to 127.8.226.66:41479: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:19:06.840634 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 11749
I20260925 15:19:06.855017 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11882
W20260925 15:19:06.857681 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11882: Not authorized: Client connection negotiation failed: client connection to 127.8.226.67:39461: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:19:06.857826 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 11882
I20260925 15:19:06.871603 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 11522
I20260925 15:19:06.872695 11584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:06.970988 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 11522
2026-09-25T15:19:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3315 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:07 dist-test-slave-8vw7 krb5kdc[12027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:07 dist-test-slave-8vw7 krb5kdc[12027](info): set up 2 sockets
Sep 25 15:19:07 dist-test-slave-8vw7 krb5kdc[12027](info): commencing operation
krb5kdc: starting...
W20260925 15:19:08.021166 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:08 dist-test-slave-8vw7 krb5kdc[12027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:08.188228 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38449
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:38449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:08.501745 12040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:08.502032 12040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:08.502182 12040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:08.508711 12040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:08.508826 12040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:08.508911 12040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:08.509027 12040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:08.509099 12040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:08.518050 12040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:38449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:08.519227 12040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:08.520583 12040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:08.531607 12046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:08.531752 12045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:08.531764 12048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.532253 12040 server_base.cc:1034] running on GCE node
I20260925 15:19:08.533003 12040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:08.534291 12040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:08.535467 12040 hybrid_clock.cc:648] HybridClock initialized: now 1790349548535428 us; error 52 us; skew 500 ppm
I20260925 15:19:08.535766 12040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:08 dist-test-slave-8vw7 krb5kdc[12027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:08.546164 12040 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:08.547617 12040 webserver.cc:460] Webserver started at http://127.8.226.126:41615/ using document root <none> and password file <none>
I20260925 15:19:08.548283 12040 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:08.548403 12040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:08.548695 12040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:08.551502 12040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "9a1e82891aea488eb036aae2c3cd9989"
format_stamp: "Formatted at 2026-09-25 15:19:08 on dist-test-slave-8vw7"
server_key: "c256f0df4ba9e8e9a0cf70d7eb0e1333"
server_key_iv: "d494332616db22e3bbc572a38f563685"
server_key_version: "encryptionkey@0"
I20260925 15:19:08.552215 12040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "9a1e82891aea488eb036aae2c3cd9989"
format_stamp: "Formatted at 2026-09-25 15:19:08 on dist-test-slave-8vw7"
server_key: "c256f0df4ba9e8e9a0cf70d7eb0e1333"
server_key_iv: "d494332616db22e3bbc572a38f563685"
server_key_version: "encryptionkey@0"
I20260925 15:19:08.556910 12040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:19:08.560101 12055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.561309 12040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:08.561462 12040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "9a1e82891aea488eb036aae2c3cd9989"
format_stamp: "Formatted at 2026-09-25 15:19:08 on dist-test-slave-8vw7"
server_key: "c256f0df4ba9e8e9a0cf70d7eb0e1333"
server_key_iv: "d494332616db22e3bbc572a38f563685"
server_key_version: "encryptionkey@0"
I20260925 15:19:08.561628 12040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:08.587692 12040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:08.591464 12040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:08.591737 12040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:08.608528 12040 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:38449
I20260925 15:19:08.608548 12106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:38449 every 8 connection(s)
I20260925 15:19:08.609951 12040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:08.613440 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:08.613708 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12040
I20260925 15:19:08.613894 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:08.614504 9097 external_mini_cluster.cc:1428] Setting key e87cdaf56183c2c38ae55afdc1243919
I20260925 15:19:08.626744 12107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989: Bootstrap starting.
Sep 25 15:19:08 dist-test-slave-8vw7 krb5kdc[12027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:08.629776 12107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:08.631080 12107 log.cc:824] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:08.634078 12107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989: No bootstrap required, opened a new log
I20260925 15:19:08.636821 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.616201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.8.226.126:38449)
0925 15:19:08.616993 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:08.617008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:08.617060 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:19:08.619593 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:08.619619 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:08.620092 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:08.620526 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:08.621925 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:08.623833 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:08.625069 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:08.626022 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:08.626187 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:08.629827 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:08.629839 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:08.629857 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:08.629896 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:08.633213 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:08.633937 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:08.633942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:08.633947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:08.634085 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:08.634446 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:08.634449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:08.634451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:08.634906 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:08.635205 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:08.635595 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 15:19:08.635942 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":312,"threads_started":1}
I20260925 15:19:08.640439 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:08.642061 12107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e82891aea488eb036aae2c3cd9989" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38449 } }
I20260925 15:19:08.642402 12107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:08.642503 12107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e82891aea488eb036aae2c3cd9989, State: Initialized, Role: FOLLOWER
I20260925 15:19:08.643062 12107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e82891aea488eb036aae2c3cd9989" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38449 } }
I20260925 15:19:08.643287 12107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:08.643414 12107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:08.643580 12107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:08.645841 12107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e82891aea488eb036aae2c3cd9989" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38449 } }
I20260925 15:19:08.646291 12107 leader_election.cc:302] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a1e82891aea488eb036aae2c3cd9989; no voters:
I20260925 15:19:08.646829 12107 leader_election.cc:288] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:08.647002 12113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:08.648623 12107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:08.649215 12113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [term 1 LEADER]: Becoming Leader. State: Replica: 9a1e82891aea488eb036aae2c3cd9989, State: Running, Role: LEADER
I20260925 15:19:08.649820 12113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e82891aea488eb036aae2c3cd9989" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38449 } }
I20260925 15:19:08.652464 12114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a1e82891aea488eb036aae2c3cd9989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e82891aea488eb036aae2c3cd9989" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38449 } } }
I20260925 15:19:08.652807 12114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:08.652503 12115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a1e82891aea488eb036aae2c3cd9989. Latest consensus state: current_term: 1 leader_uuid: "9a1e82891aea488eb036aae2c3cd9989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e82891aea488eb036aae2c3cd9989" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38449 } } }
I20260925 15:19:08.652915 12115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:08.664258 12120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:08.669616 12120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:08.687567 12120 catalog_manager.cc:1347] Generated new cluster ID: 15bcc41c90424c49b5b3a7c41f499ca0
I20260925 15:19:08.687676 12120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:08.700018 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.688196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.8.226.126:38449)
0925 15:19:08.688373 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:08.688382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:08.688988 (+ 606us) server_negotiation.cc:407] Connection header received
0925 15:19:08.689228 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:08.689235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:08.689335 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:08.689461 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:08.691209 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:08.692558 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:08.694866 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:08.695711 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:08.695839 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:08.697448 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:08.697456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:08.697464 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:08.698552 (+ 1088us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:08.698671 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:08.698711 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:08.698810 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:08.699078 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 15:19:08.699203 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:19:08.703485 12120 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:08.704594 12120 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:08.711542 12120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a1e82891aea488eb036aae2c3cd9989: Generated new TSK 0
I20260925 15:19:08.712285 12120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:08.721804 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12040
I20260925 15:19:08.723616 12102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:08.806927 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12040
2026-09-25T15:19:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1827 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:08 dist-test-slave-8vw7 krb5kdc[12141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:08 dist-test-slave-8vw7 krb5kdc[12141](info): set up 2 sockets
Sep 25 15:19:08 dist-test-slave-8vw7 krb5kdc[12141](info): commencing operation
krb5kdc: starting...
W20260925 15:19:09.837432 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:09 dist-test-slave-8vw7 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:10.012207 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:33387
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:10.317065 12154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:10.317297 12154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:10.317371 12154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:10.324256 12154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:10.324352 12154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:10.324394 12154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:10.324451 12154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:10.324493 12154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:10.333616 12154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:10.334681 12154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:10.335830 12154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:10.346474 12159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:10.346436 12162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:10.346587 12160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:10.347437 12154 server_base.cc:1034] running on GCE node
I20260925 15:19:10.348206 12154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:10.349526 12154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:10.350759 12154 hybrid_clock.cc:648] HybridClock initialized: now 1790349550350714 us; error 55 us; skew 500 ppm
I20260925 15:19:10.351071 12154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:10 dist-test-slave-8vw7 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349550, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:10.353902 12154 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:10.355167 12154 webserver.cc:460] Webserver started at http://127.8.226.126:36647/ using document root <none> and password file <none>
I20260925 15:19:10.355786 12154 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:10.355950 12154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:10.356241 12154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:10.359148 12154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "4fb7c36ce93a4299b8bc6b77b745158c"
format_stamp: "Formatted at 2026-09-25 15:19:10 on dist-test-slave-8vw7"
server_key: "69dc152d3f692fad4c8b2f85421114ad"
server_key_iv: "407e22ffb8c9b5a750ae00d9358c2980"
server_key_version: "encryptionkey@0"
I20260925 15:19:10.359866 12154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "4fb7c36ce93a4299b8bc6b77b745158c"
format_stamp: "Formatted at 2026-09-25 15:19:10 on dist-test-slave-8vw7"
server_key: "69dc152d3f692fad4c8b2f85421114ad"
server_key_iv: "407e22ffb8c9b5a750ae00d9358c2980"
server_key_version: "encryptionkey@0"
I20260925 15:19:10.364450 12154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:19:10.367578 12169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:10.368723 12154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:10.368906 12154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "4fb7c36ce93a4299b8bc6b77b745158c"
format_stamp: "Formatted at 2026-09-25 15:19:10 on dist-test-slave-8vw7"
server_key: "69dc152d3f692fad4c8b2f85421114ad"
server_key_iv: "407e22ffb8c9b5a750ae00d9358c2980"
server_key_version: "encryptionkey@0"
I20260925 15:19:10.369133 12154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:10.394598 12154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:10.399016 12154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:10.399298 12154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:10.416085 12154 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:33387
I20260925 15:19:10.416116 12220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:33387 every 8 connection(s)
I20260925 15:19:10.417419 12154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:10.420835 12221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:10.427861 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12154
I20260925 15:19:10.428083 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:10.428683 9097 external_mini_cluster.cc:1428] Setting key 43f63f071543058766a105af683b3e87
I20260925 15:19:10.430389 12221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c: Bootstrap starting.
I20260925 15:19:10.433267 12221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:10.434466 12221 log.cc:824] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:10.437266 12221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c: No bootstrap required, opened a new log
Sep 25 15:19:10 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:10.447199 12221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33387 } }
I20260925 15:19:10.447623 12221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:10.447729 12221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb7c36ce93a4299b8bc6b77b745158c, State: Initialized, Role: FOLLOWER
I20260925 15:19:10.448320 12221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33387 } }
I20260925 15:19:10.448590 12221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:10.448750 12221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:10.448930 12221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:10.451283 12221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33387 } }
I20260925 15:19:10.451613 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:10.430620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.8.226.126:33387)
0925 15:19:10.431352 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:10.431362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:10.431398 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:19:10.432937 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:10.433021 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:10.433358 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:10.433701 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:10.435391 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:10.436715 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:10.438488 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:10.439135 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:10.439269 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:10.443645 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:10.443655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:10.443673 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:10.443716 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:10.447460 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:10.448279 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:10.448288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:10.448299 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:10.448496 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:10.448947 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:10.448972 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:10.448977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:10.449531 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:10.449904 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:10.450311 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 15:19:10.450682 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":367,"threads_started":1}
I20260925 15:19:10.451759 12221 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fb7c36ce93a4299b8bc6b77b745158c; no voters:
I20260925 15:19:10.452421 12221 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:10.452523 12226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:10.453171 12226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [term 1 LEADER]: Becoming Leader. State: Replica: 4fb7c36ce93a4299b8bc6b77b745158c, State: Running, Role: LEADER
I20260925 15:19:10.453949 12226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33387 } }
I20260925 15:19:10.455921 12221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:10.456614 12228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33387 } } }
I20260925 15:19:10.456941 12228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:10.457257 12227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb7c36ce93a4299b8bc6b77b745158c. Latest consensus state: current_term: 1 leader_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb7c36ce93a4299b8bc6b77b745158c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33387 } } }
I20260925 15:19:10.457751 12227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c [sys.catalog]: This master's current role is: LEADER
W20260925 15:19:10.466158 12242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:10.466275 12242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:10.466404 12231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:10.468070 12231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:10.475623 12231 catalog_manager.cc:1347] Generated new cluster ID: 7465c2c2bf064a7e887313cccdc2a2fd
I20260925 15:19:10.475739 12231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:10.497406 12231 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:10.499050 12231 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:10.510635 12231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb7c36ce93a4299b8bc6b77b745158c: Generated new TSK 0
I20260925 15:19:10.511382 12231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:19:10.562531 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33387
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:10.882169 12247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:10.882400 12247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:10.882483 12247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:10.889071 12247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:10.889155 12247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:10.889258 12247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:10.897799 12247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:10.898922 12247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:10.900118 12247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:10.912181 12252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:10.912142 12255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:10.912210 12253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:10.912714 12247 server_base.cc:1034] running on GCE node
I20260925 15:19:10.913291 12247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:10.914110 12247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:10.915313 12247 hybrid_clock.cc:648] HybridClock initialized: now 1790349550915278 us; error 53 us; skew 500 ppm
I20260925 15:19:10.915612 12247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:10 dist-test-slave-8vw7 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349550, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:10.918573 12247 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:19:10.919775 12247 webserver.cc:460] Webserver started at http://127.8.226.65:34035/ using document root <none> and password file <none>
I20260925 15:19:10.920351 12247 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:10.920478 12247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:10.920744 12247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:10.923398 12247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "69d4305dcce44127a2aedbf803250225"
format_stamp: "Formatted at 2026-09-25 15:19:10 on dist-test-slave-8vw7"
server_key: "f482e4a4f6f991836349d3b15031d8e4"
server_key_iv: "d07d27b2cedb40e377eaa4e16efa4f2a"
server_key_version: "encryptionkey@0"
I20260925 15:19:10.924101 12247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "69d4305dcce44127a2aedbf803250225"
format_stamp: "Formatted at 2026-09-25 15:19:10 on dist-test-slave-8vw7"
server_key: "f482e4a4f6f991836349d3b15031d8e4"
server_key_iv: "d07d27b2cedb40e377eaa4e16efa4f2a"
server_key_version: "encryptionkey@0"
I20260925 15:19:10.928480 12247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:19:10.931592 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:10.932731 12247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:19:10.932888 12247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "69d4305dcce44127a2aedbf803250225"
format_stamp: "Formatted at 2026-09-25 15:19:10 on dist-test-slave-8vw7"
server_key: "f482e4a4f6f991836349d3b15031d8e4"
server_key_iv: "d07d27b2cedb40e377eaa4e16efa4f2a"
server_key_version: "encryptionkey@0"
I20260925 15:19:10.933074 12247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:10.946655 12247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:10.950387 12247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:10.950708 12247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:10.951566 12247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:10.952620 12247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:10.952708 12247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:10.952831 12247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:10.952894 12247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:10.983933 12247 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:44289
I20260925 15:19:10.984009 12374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:44289 every 8 connection(s)
I20260925 15:19:10.985190 12247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:10.989930 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12247
I20260925 15:19:10.990119 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:10.990774 9097 external_mini_cluster.cc:1428] Setting key dea8ce8edcd3bba94963f99b7a1bf2ce
Sep 25 15:19:10 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349550, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:11.004141 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:10.987491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:39045 (local address 127.8.226.126:33387)
0925 15:19:10.987987 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:10.987993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:10.988929 (+ 936us) server_negotiation.cc:407] Connection header received
0925 15:19:10.990714 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:10.990720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:10.990808 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:10.990930 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:10.992261 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:10.993058 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:10.994521 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:10.994896 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:10.995019 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:10.998710 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:10.998714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:10.998717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:10.998746 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:11.001158 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.002016 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.002020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.002022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.002129 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.002583 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.002585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.002587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.002824 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:11.002924 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:11.003755 (+ 831us) server_negotiation.cc:299] Negotiation successful
0925 15:19:11.003895 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":226,"threads_started":1}
I20260925 15:19:11.005498 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:10.988108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33387 (local address 127.8.226.65:39045)
0925 15:19:10.988768 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:10.988805 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:10.990466 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:10.991125 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:10.991132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:10.991576 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:10.992083 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:10.992098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:10.993201 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:10.993206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:10.994377 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:10.994388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:10.995107 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:10.995117 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:10.995353 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:10.995975 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:10.995993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:10.998522 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:11.001279 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:11.001290 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:11.001307 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:11.001886 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:11.002227 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:11.002235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:11.002241 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:11.002477 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:11.002941 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:11.002951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:11.003400 (+ 449us) client_negotiation.cc:769] Sending connection context
0925 15:19:11.003718 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 15:19:11.004239 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
I20260925 15:19:11.006978 12375 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33387
I20260925 15:19:11.007304 12375 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:11.008167 12375 heartbeater.cc:511] Master 127.8.226.126:33387 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:19:11.010445 12186 ts_manager.cc:194] Registered new tserver with Master: 69d4305dcce44127a2aedbf803250225 (127.8.226.65:44289)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
I20260925 15:19:11.012322 12186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:39045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:19:11.032142 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33387
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:11.327034 12381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:11.327379 12381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:11.327474 12381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:11.334030 12381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:11.334115 12381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:11.334215 12381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:11.343084 12381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:11.344110 12381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:11.345328 12381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:11.357211 12386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:11.357200 12387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:11.357229 12389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:11.358109 12381 server_base.cc:1034] running on GCE node
I20260925 15:19:11.358712 12381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:11.359594 12381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:11.360816 12381 hybrid_clock.cc:648] HybridClock initialized: now 1790349551360787 us; error 48 us; skew 500 ppm
I20260925 15:19:11.361117 12381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:11.364053 12381 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:19:11.365357 12381 webserver.cc:460] Webserver started at http://127.8.226.66:39813/ using document root <none> and password file <none>
I20260925 15:19:11.365959 12381 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:11.366096 12381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:11.366384 12381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:11.369180 12381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "07743ccb9abf439eadb804de8ce9aee3"
format_stamp: "Formatted at 2026-09-25 15:19:11 on dist-test-slave-8vw7"
server_key: "47e49aba699df48453b39012dc5fcae7"
server_key_iv: "8dd46f5070a373a835aaf717e99edb51"
server_key_version: "encryptionkey@0"
I20260925 15:19:11.369959 12381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "07743ccb9abf439eadb804de8ce9aee3"
format_stamp: "Formatted at 2026-09-25 15:19:11 on dist-test-slave-8vw7"
server_key: "47e49aba699df48453b39012dc5fcae7"
server_key_iv: "8dd46f5070a373a835aaf717e99edb51"
server_key_version: "encryptionkey@0"
I20260925 15:19:11.374626 12381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:19:11.377667 12396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:11.378835 12381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:11.379009 12381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "07743ccb9abf439eadb804de8ce9aee3"
format_stamp: "Formatted at 2026-09-25 15:19:11 on dist-test-slave-8vw7"
server_key: "47e49aba699df48453b39012dc5fcae7"
server_key_iv: "8dd46f5070a373a835aaf717e99edb51"
server_key_version: "encryptionkey@0"
I20260925 15:19:11.379204 12381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:11.402356 12381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:11.406023 12381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:11.406280 12381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:11.407087 12381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:11.408341 12381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:11.408406 12381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:11.408550 12381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:11.408597 12381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:11.441565 12381 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:45079
I20260925 15:19:11.441609 12508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:45079 every 8 connection(s)
I20260925 15:19:11.442721 12381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:11.449239 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12381
I20260925 15:19:11.449410 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:11.449842 9097 external_mini_cluster.cc:1428] Setting key 6dceb09043b7deae7999ba38f675e0cd
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:11.461254 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.445013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:50887 (local address 127.8.226.126:33387)
0925 15:19:11.445212 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:11.445217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:11.446233 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 15:19:11.447925 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:11.447931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:11.448014 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:11.448132 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:11.449679 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.450563 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.452283 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.452655 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.452777 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:11.455789 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:11.455795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:11.455799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:11.455835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:11.458518 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.459201 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.459206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.459210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.459319 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.459699 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.459704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.459707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.459946 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:11.460113 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:11.460689 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:19:11.460898 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:19:11.462080 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.445480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33387 (local address 127.8.226.66:50887)
0925 15:19:11.446095 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:11.446132 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:11.447650 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:11.448349 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:11.448357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:11.448836 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:11.449527 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:11.449543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.450732 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.450737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:11.452119 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:11.452147 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.452848 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.452856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:11.453123 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:11.453725 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:11.453744 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:11.455594 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:11.458655 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:11.458663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:11.458677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:11.459080 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:11.459417 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:11.459422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:11.459425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:11.459583 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:11.460082 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:11.460089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:11.460376 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 15:19:11.460602 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 15:19:11.461100 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":246,"threads_started":1}
I20260925 15:19:11.463469 12509 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33387
I20260925 15:19:11.463737 12509 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:11.464406 12509 heartbeater.cc:511] Master 127.8.226.126:33387 requested a full tablet report, sending...
I20260925 15:19:11.465727 12186 ts_manager.cc:194] Registered new tserver with Master: 07743ccb9abf439eadb804de8ce9aee3 (127.8.226.66:45079)
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:19:11.466511 12186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:50887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:19:11.487923 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33387
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:11.784251 12514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:11.784500 12514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:11.784580 12514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:11.791361 12514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:11.791460 12514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:11.791563 12514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:11.800149 12514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:11.801185 12514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:11.802347 12514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:11.814530 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:11.814640 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:11.814666 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:11.815384 12514 server_base.cc:1034] running on GCE node
I20260925 15:19:11.815846 12514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:11.816576 12514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:11.817814 12514 hybrid_clock.cc:648] HybridClock initialized: now 1790349551817752 us; error 31 us; skew 500 ppm
I20260925 15:19:11.818120 12514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:11.821043 12514 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:19:11.822292 12514 webserver.cc:460] Webserver started at http://127.8.226.67:38073/ using document root <none> and password file <none>
I20260925 15:19:11.822918 12514 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:11.823045 12514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:11.823326 12514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:11.825985 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "48261bc7fa5c4e5fa1e08b90e80b3b37"
format_stamp: "Formatted at 2026-09-25 15:19:11 on dist-test-slave-8vw7"
server_key: "58782ebfc9ea5c79b68564d53f4eedc3"
server_key_iv: "fc6badb4b22b37459738886b1a34da88"
server_key_version: "encryptionkey@0"
I20260925 15:19:11.826723 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "48261bc7fa5c4e5fa1e08b90e80b3b37"
format_stamp: "Formatted at 2026-09-25 15:19:11 on dist-test-slave-8vw7"
server_key: "58782ebfc9ea5c79b68564d53f4eedc3"
server_key_iv: "fc6badb4b22b37459738886b1a34da88"
server_key_version: "encryptionkey@0"
I20260925 15:19:11.831528 12514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:19:11.834769 12529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:11.836064 12514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:11.836226 12514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "48261bc7fa5c4e5fa1e08b90e80b3b37"
format_stamp: "Formatted at 2026-09-25 15:19:11 on dist-test-slave-8vw7"
server_key: "58782ebfc9ea5c79b68564d53f4eedc3"
server_key_iv: "fc6badb4b22b37459738886b1a34da88"
server_key_version: "encryptionkey@0"
I20260925 15:19:11.836400 12514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:11.850526 12514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:11.854048 12514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:11.854305 12514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:11.855156 12514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:11.856367 12514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:11.856462 12514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:11.856596 12514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:11.856673 12514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:11.888437 12514 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:35809
I20260925 15:19:11.888479 12641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:35809 every 8 connection(s)
I20260925 15:19:11.889748 12514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:11.892825 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12514
I20260925 15:19:11.893009 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:11.893560 9097 external_mini_cluster.cc:1428] Setting key 72520495e3c076539caf4eff1564c7e9
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:11.907523 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.892108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:51105 (local address 127.8.226.126:33387)
0925 15:19:11.892294 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:11.892298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:11.893584 (+ 1286us) server_negotiation.cc:407] Connection header received
0925 15:19:11.895600 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:11.895606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:11.895686 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:11.895806 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:11.897246 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.897942 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.899303 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.899664 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.899752 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:11.902554 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:11.902558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:11.902560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:11.902586 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:11.904999 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.905605 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.905608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.905611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.905722 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.906078 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.906081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.906083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.906343 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:11.906438 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:11.907061 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 15:19:11.907293 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:19:11.908561 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.892613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33387 (local address 127.8.226.67:51105)
0925 15:19:11.893397 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:11.893438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:11.895329 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:11.895992 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:11.896000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:11.896528 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:11.897112 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:11.897130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.898059 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.898063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:11.899197 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:11.899207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.899837 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.899841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:11.900005 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:11.900527 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:11.900546 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:11.902406 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:11.905133 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:11.905139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:11.905148 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:11.905504 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:11.905814 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:11.905816 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:11.905818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:11.905987 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:11.906430 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:11.906435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:11.906692 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 15:19:11.906946 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 15:19:11.907405 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":351,"threads_started":1}
I20260925 15:19:11.909765 12642 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33387
I20260925 15:19:11.910013 12642 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:11.910615 12642 heartbeater.cc:511] Master 127.8.226.126:33387 requested a full tablet report, sending...
I20260925 15:19:11.911804 12186 ts_manager.cc:194] Registered new tserver with Master: 48261bc7fa5c4e5fa1e08b90e80b3b37 (127.8.226.67:35809)
I20260925 15:19:11.912577 12186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:51105
I20260925 15:19:11.919999 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:11.964265 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.950876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.8.226.126:33387)
0925 15:19:11.951100 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:11.951105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:11.951379 (+ 274us) server_negotiation.cc:407] Connection header received
0925 15:19:11.951516 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:11.951520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:11.951576 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:11.951655 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:11.953192 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.953865 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.955211 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.955560 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.955643 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:11.959518 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:11.959522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:11.959525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:11.959552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:11.961930 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.962436 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.962439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.962442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.962564 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.962963 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.962969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.962972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.963254 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:11.963493 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:11.963780 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:19:11.963985 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 15:19:11.964826 12186 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32894
I20260925 15:19:11.966614 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12247
Sep 25 15:19:11 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:11.981288 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.967656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.8.226.65:44289)
0925 15:19:11.968157 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:11.968165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:11.968188 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:19:11.968252 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:11.968259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:11.968450 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:11.968572 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:11.969702 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.970648 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.971730 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:11.972258 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:11.972358 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:11.975895 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:11.975905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:11.975918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:11.975947 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:11.978726 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.979373 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.979380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.979382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.979494 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:11.979843 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:11.979850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:11.979853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:11.980155 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:11.980452 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:11.980710 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:19:11.980953 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
W20260925 15:19:11.982524 12370 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43314
W20260925 15:19:11.983676 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:19:11.983923 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12247
I20260925 15:19:11.998564 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12381
Sep 25 15:19:12 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:19:12.012306 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.999337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.8.226.66:45079)
0925 15:19:11.999827 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:11.999839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:11.999860 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:19:11.999914 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:11.999919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:12.000108 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:12.000221 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:12.001306 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:12.002195 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:12.003224 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:12.003709 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:12.003792 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:12.007363 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:12.007371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:12.007383 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:12.007416 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:12.010184 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:12.010713 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:12.010719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:12.010722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:12.010833 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:12.011130 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:12.011133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:12.011135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:12.011431 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:12.011641 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:12.011835 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:19:12.012000 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":258,"threads_started":1}
W20260925 15:19:12.013589 12504 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33782
W20260925 15:19:12.014555 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12381: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:19:12.014698 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12381
I20260925 15:19:12.028522 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12514
Sep 25 15:19:12 dist-test-slave-8vw7 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349551, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:19:12.043256 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:12.029410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.8.226.67:35809)
0925 15:19:12.030001 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:12.030008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:12.030033 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:12.030083 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:12.030089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:12.030281 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:12.030443 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:12.031622 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:12.032548 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:12.033717 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:12.034206 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:12.034338 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:12.038306 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:12.038315 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:12.038328 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:12.038356 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:12.041056 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:12.041627 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:12.041633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:12.041635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:12.041744 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:12.042043 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:12.042046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:12.042048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:12.042346 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:12.042572 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:12.042801 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 15:19:12.042976 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":299,"threads_started":1}
W20260925 15:19:12.044392 12637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45192
W20260925 15:19:12.045483 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12514: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:19:12.045608 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12514
I20260925 15:19:12.060290 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12154
I20260925 15:19:12.061409 12216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:12.160815 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12154
2026-09-25T15:19:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3361 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:12 dist-test-slave-8vw7 krb5kdc[12659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:12 dist-test-slave-8vw7 krb5kdc[12659](info): set up 2 sockets
Sep 25 15:19:12 dist-test-slave-8vw7 krb5kdc[12659](info): commencing operation
krb5kdc: starting...
W20260925 15:19:13.193220 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:13 dist-test-slave-8vw7 krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:13.372254 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:42037
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:13.670439 12672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:13.670694 12672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:13.670868 12672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:13.677448 12672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:13.677560 12672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:13.677634 12672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:13.677731 12672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:13.677799 12672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:13.686509 12672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:13.687670 12672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:13.688932 12672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:13.700021 12677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:13.700059 12678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:13.700109 12680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:13.700912 12672 server_base.cc:1034] running on GCE node
I20260925 15:19:13.701700 12672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:13.702911 12672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:13.704134 12672 hybrid_clock.cc:648] HybridClock initialized: now 1790349553704104 us; error 41 us; skew 500 ppm
I20260925 15:19:13.704460 12672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:13 dist-test-slave-8vw7 krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349553, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:13.711015 12672 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:13.712322 12672 webserver.cc:460] Webserver started at http://127.8.226.126:43597/ using document root <none> and password file <none>
I20260925 15:19:13.713020 12672 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:13.713155 12672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:13.713452 12672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:13.716220 12672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "f62bb92706a74a92a2fd8c2ca99c415b"
format_stamp: "Formatted at 2026-09-25 15:19:13 on dist-test-slave-8vw7"
server_key: "7c61a9d0d0bc193f49907a68480c60fe"
server_key_iv: "7e3a1a48be637d8b551cf6e78e419592"
server_key_version: "encryptionkey@0"
I20260925 15:19:13.716997 12672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "f62bb92706a74a92a2fd8c2ca99c415b"
format_stamp: "Formatted at 2026-09-25 15:19:13 on dist-test-slave-8vw7"
server_key: "7c61a9d0d0bc193f49907a68480c60fe"
server_key_iv: "7e3a1a48be637d8b551cf6e78e419592"
server_key_version: "encryptionkey@0"
I20260925 15:19:13.721666 12672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:13.724932 12687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:13.726225 12672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:19:13.726392 12672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "f62bb92706a74a92a2fd8c2ca99c415b"
format_stamp: "Formatted at 2026-09-25 15:19:13 on dist-test-slave-8vw7"
server_key: "7c61a9d0d0bc193f49907a68480c60fe"
server_key_iv: "7e3a1a48be637d8b551cf6e78e419592"
server_key_version: "encryptionkey@0"
I20260925 15:19:13.726585 12672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:13.742805 12672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:13.746843 12672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:13.747143 12672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:13.764636 12672 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:42037
I20260925 15:19:13.764663 12738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:42037 every 8 connection(s)
I20260925 15:19:13.765934 12672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:13.768898 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12672
I20260925 15:19:13.769156 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:13.769780 12739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:13.769958 9097 external_mini_cluster.cc:1428] Setting key 564b83fafa96331563ba504262264ad4
I20260925 15:19:13.780879 12739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b: Bootstrap starting.
Sep 25 15:19:13 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:13.784081 12739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:13.785252 12739 log.cc:824] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:13.788228 12739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b: No bootstrap required, opened a new log
I20260925 15:19:13.790473 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:13.771815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.8.226.126:42037)
0925 15:19:13.772505 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:13.772515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:13.772556 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:19:13.774234 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:13.774254 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:13.774599 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:13.774960 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:13.776292 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:13.777482 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:13.778619 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:13.779276 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:13.779469 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:13.783341 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:13.783347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:13.783359 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:13.783386 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:13.786699 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:13.787379 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:13.787385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:13.787390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:13.787526 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:13.787977 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:13.787980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:13.787982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:13.788485 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:13.788721 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:13.789221 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 15:19:13.789528 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":309,"threads_started":1}
I20260925 15:19:13.795523 12739 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42037 } }
I20260925 15:19:13.795830 12739 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:13.795931 12739 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62bb92706a74a92a2fd8c2ca99c415b, State: Initialized, Role: FOLLOWER
I20260925 15:19:13.796473 12739 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42037 } }
I20260925 15:19:13.796672 12739 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:13.796790 12739 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:13.796938 12739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:13.798897 12739 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42037 } }
I20260925 15:19:13.799391 12739 leader_election.cc:302] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62bb92706a74a92a2fd8c2ca99c415b; no voters:
I20260925 15:19:13.799860 12739 leader_election.cc:288] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:13.799969 12744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:13.800565 12744 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [term 1 LEADER]: Becoming Leader. State: Replica: f62bb92706a74a92a2fd8c2ca99c415b, State: Running, Role: LEADER
I20260925 15:19:13.801069 12744 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42037 } }
I20260925 15:19:13.801733 12739 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:13.803586 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62bb92706a74a92a2fd8c2ca99c415b. Latest consensus state: current_term: 1 leader_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42037 } } }
I20260925 15:19:13.804023 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:13.804384 12745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62bb92706a74a92a2fd8c2ca99c415b" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42037 } } }
I20260925 15:19:13.804694 12745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:13.806300 12753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:13.810580 12753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:13.817786 12753 catalog_manager.cc:1347] Generated new cluster ID: beced38706174129a0cbc301fd5d4d46
I20260925 15:19:13.817890 12753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:13.838562 12753 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:13.840376 12753 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:13.851923 12753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f62bb92706a74a92a2fd8c2ca99c415b: Generated new TSK 0
I20260925 15:19:13.852979 12753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:19:13.898595 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:42037
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:14.206161 12765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:14.206408 12765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:14.206493 12765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:14.213348 12765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:14.213439 12765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:14.213533 12765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:14.222337 12765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:14.223393 12765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:14.224622 12765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:14.237092 12773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:14.237051 12771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:14.237586 12770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:14.237931 12765 server_base.cc:1034] running on GCE node
I20260925 15:19:14.238492 12765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:14.239346 12765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:14.240554 12765 hybrid_clock.cc:648] HybridClock initialized: now 1790349554240516 us; error 47 us; skew 500 ppm
I20260925 15:19:14.240871 12765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:14 dist-test-slave-8vw7 krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349554, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:14.243604 12765 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:19:14.245024 12765 webserver.cc:460] Webserver started at http://127.8.226.65:33361/ using document root <none> and password file <none>
I20260925 15:19:14.245613 12765 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:14.245748 12765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:14.246021 12765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:14.248664 12765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "d882f0b7563d47b2a665803ac04c99f3"
format_stamp: "Formatted at 2026-09-25 15:19:14 on dist-test-slave-8vw7"
server_key: "7eaeceb7f317336aae0f9996ef8bc771"
server_key_iv: "45b4bc2c5708e7c17bfd59d9b21cd6e4"
server_key_version: "encryptionkey@0"
I20260925 15:19:14.249445 12765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "d882f0b7563d47b2a665803ac04c99f3"
format_stamp: "Formatted at 2026-09-25 15:19:14 on dist-test-slave-8vw7"
server_key: "7eaeceb7f317336aae0f9996ef8bc771"
server_key_iv: "45b4bc2c5708e7c17bfd59d9b21cd6e4"
server_key_version: "encryptionkey@0"
I20260925 15:19:14.254086 12765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:19:14.257305 12780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:14.258534 12765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:14.258713 12765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "d882f0b7563d47b2a665803ac04c99f3"
format_stamp: "Formatted at 2026-09-25 15:19:14 on dist-test-slave-8vw7"
server_key: "7eaeceb7f317336aae0f9996ef8bc771"
server_key_iv: "45b4bc2c5708e7c17bfd59d9b21cd6e4"
server_key_version: "encryptionkey@0"
I20260925 15:19:14.258893 12765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:14.279599 12765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:14.282815 12765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:14.283058 12765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:14.283912 12765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:14.285060 12765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:14.285172 12765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:14.285257 12765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:14.285296 12765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:14.315765 12765 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:45927
I20260925 15:19:14.315815 12892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:45927 every 8 connection(s)
I20260925 15:19:14.316844 12765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:14.326022 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12765
I20260925 15:19:14.326248 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:14.326947 9097 external_mini_cluster.cc:1428] Setting key 5484e49dd93d19408425b3bcc5a1ed5b
Sep 25 15:19:14 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349554, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:14.336730 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.319121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:40189 (local address 127.8.226.126:42037)
0925 15:19:14.319550 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:14.319554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:14.320407 (+ 853us) server_negotiation.cc:407] Connection header received
0925 15:19:14.322179 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:14.322185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:14.322268 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:14.322376 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:14.323758 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.324452 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.326277 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.326704 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.326801 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:14.331046 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:14.331051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:14.331054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:14.331083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:14.333652 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:14.334411 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:14.334418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:14.334423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:14.334570 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:14.334956 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:14.334963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:14.334967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:14.335326 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:14.335465 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:14.336035 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 15:19:14.336283 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":242,"threads_started":1}
I20260925 15:19:14.337229 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.319566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:42037 (local address 127.8.226.65:40189)
0925 15:19:14.320258 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:14.320299 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:14.321934 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:14.322579 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:14.322587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:14.323054 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:14.323575 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:14.323591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.324599 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.324603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:14.326139 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:14.326154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.326915 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.326922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:14.327207 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:14.327906 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:14.327964 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:14.330866 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:14.333796 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:14.333807 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:14.333824 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:14.334291 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:14.334672 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:14.334675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:14.334677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:14.334840 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:14.335431 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:14.335436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:14.335693 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 15:19:14.335890 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 15:19:14.336231 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":283,"threads_started":1}
I20260925 15:19:14.338621 12893 heartbeater.cc:347] Connected to a master server at 127.8.226.126:42037
I20260925 15:19:14.338914 12893 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:14.339540 12893 heartbeater.cc:511] Master 127.8.226.126:42037 requested a full tablet report, sending...
I20260925 15:19:14.341513 12704 ts_manager.cc:194] Registered new tserver with Master: d882f0b7563d47b2a665803ac04c99f3 (127.8.226.65:45927)
I20260925 15:19:14.342785 12704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:40189
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:19:14.366087 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:42037
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:14.668638 12899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:14.668893 12899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:14.669068 12899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:14.675629 12899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:14.675748 12899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:14.675928 12899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:14.684517 12899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:14.685599 12899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:14.686775 12899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:14.698881 12904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:14.698808 12905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:14.698925 12907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:14.699417 12899 server_base.cc:1034] running on GCE node
I20260925 15:19:14.700034 12899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:14.700944 12899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:14.702208 12899 hybrid_clock.cc:648] HybridClock initialized: now 1790349554702150 us; error 53 us; skew 500 ppm
I20260925 15:19:14.702541 12899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:14 dist-test-slave-8vw7 krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349554, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:14.705423 12899 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:19:14.706727 12899 webserver.cc:460] Webserver started at http://127.8.226.66:42225/ using document root <none> and password file <none>
I20260925 15:19:14.707325 12899 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:14.707415 12899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:14.707665 12899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:14.710435 12899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "2190475e869a457fab9d67fa950e98b2"
format_stamp: "Formatted at 2026-09-25 15:19:14 on dist-test-slave-8vw7"
server_key: "82a1d528dc254c05f18de5eaa6d5517f"
server_key_iv: "b380dd9980d533e55413714f472f4f3a"
server_key_version: "encryptionkey@0"
I20260925 15:19:14.711094 12899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "2190475e869a457fab9d67fa950e98b2"
format_stamp: "Formatted at 2026-09-25 15:19:14 on dist-test-slave-8vw7"
server_key: "82a1d528dc254c05f18de5eaa6d5517f"
server_key_iv: "b380dd9980d533e55413714f472f4f3a"
server_key_version: "encryptionkey@0"
I20260925 15:19:14.715688 12899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:19:14.718751 12914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:14.719966 12899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:14.720113 12899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "2190475e869a457fab9d67fa950e98b2"
format_stamp: "Formatted at 2026-09-25 15:19:14 on dist-test-slave-8vw7"
server_key: "82a1d528dc254c05f18de5eaa6d5517f"
server_key_iv: "b380dd9980d533e55413714f472f4f3a"
server_key_version: "encryptionkey@0"
I20260925 15:19:14.720304 12899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:14.741628 12899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:14.745195 12899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:14.745504 12899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:14.746374 12899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:14.747539 12899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:14.747609 12899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:14.747684 12899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:14.747725 12899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:14.779137 12899 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:44443
I20260925 15:19:14.779218 13026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:44443 every 8 connection(s)
I20260925 15:19:14.780283 12899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:14.780903 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 12899
I20260925 15:19:14.781155 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:14.781816 9097 external_mini_cluster.cc:1428] Setting key a88bff02f60f662fdba7cfc08cff7b55
Sep 25 15:19:14 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349554, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:14.800302 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.783059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:48273 (local address 127.8.226.126:42037)
0925 15:19:14.783799 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:14.783806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:14.784404 (+ 598us) server_negotiation.cc:407] Connection header received
0925 15:19:14.786130 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:14.786135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:14.786213 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:14.786331 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:14.787696 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.788422 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.789977 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.790714 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.790850 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:14.793954 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:14.793962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:14.793970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:14.794010 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:14.797303 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:14.797953 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:14.797961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:14.797967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:14.798157 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:14.798552 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:14.798557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:14.798560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:14.798895 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:14.799075 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:14.799694 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 15:19:14.799908 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585}
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:19:14.800904 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.783460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:42037 (local address 127.8.226.66:48273)
0925 15:19:14.784266 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:14.784313 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:14.785898 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:14.786556 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:14.786565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:14.787030 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:14.787536 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:14.787556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.788562 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.788574 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:14.789835 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:14.789847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:14.790899 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:14.790904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:14.791093 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:14.791722 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:14.791744 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:14.793751 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:14.797419 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:14.797425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:14.797434 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:14.797823 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:14.798270 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:14.798273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:14.798275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:14.798434 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:14.799017 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:14.799022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:14.799347 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 15:19:14.799553 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 15:19:14.799873 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":367,"threads_started":1}
I20260925 15:19:14.802234 13027 heartbeater.cc:347] Connected to a master server at 127.8.226.126:42037
I20260925 15:19:14.802533 13027 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
I20260925 15:19:14.803242 13027 heartbeater.cc:511] Master 127.8.226.126:42037 requested a full tablet report, sending...
I20260925 15:19:14.804513 12704 ts_manager.cc:194] Registered new tserver with Master: 2190475e869a457fab9d67fa950e98b2 (127.8.226.66:44443)
I20260925 15:19:14.805411 12704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:48273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:19:14.824302 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:42037
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:15.123797 13032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:15.124048 13032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:15.124214 13032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:15.131171 13032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:15.131283 13032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:15.131460 13032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:15.140379 13032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:15.141526 13032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:15.142724 13032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:15.155951 13038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:15.156044 13037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:15.155993 13040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:15.156788 13032 server_base.cc:1034] running on GCE node
I20260925 15:19:15.157380 13032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:15.158201 13032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:15.159420 13032 hybrid_clock.cc:648] HybridClock initialized: now 1790349555159370 us; error 43 us; skew 500 ppm
I20260925 15:19:15.159713 13032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:15.163092 13032 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:19:15.164355 13032 webserver.cc:460] Webserver started at http://127.8.226.67:38019/ using document root <none> and password file <none>
I20260925 15:19:15.165048 13032 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:15.165166 13032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:15.165454 13032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:15.168169 13032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "847359ed98ef48a5b8b7a2025f8f238d"
format_stamp: "Formatted at 2026-09-25 15:19:15 on dist-test-slave-8vw7"
server_key: "194ae4f74e0b6b84571c4b9686aab282"
server_key_iv: "c7a0017811f00b36437cecb39d695123"
server_key_version: "encryptionkey@0"
I20260925 15:19:15.168915 13032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "847359ed98ef48a5b8b7a2025f8f238d"
format_stamp: "Formatted at 2026-09-25 15:19:15 on dist-test-slave-8vw7"
server_key: "194ae4f74e0b6b84571c4b9686aab282"
server_key_iv: "c7a0017811f00b36437cecb39d695123"
server_key_version: "encryptionkey@0"
I20260925 15:19:15.173609 13032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:19:15.176716 13047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:15.177889 13032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:15.178040 13032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "847359ed98ef48a5b8b7a2025f8f238d"
format_stamp: "Formatted at 2026-09-25 15:19:15 on dist-test-slave-8vw7"
server_key: "194ae4f74e0b6b84571c4b9686aab282"
server_key_iv: "c7a0017811f00b36437cecb39d695123"
server_key_version: "encryptionkey@0"
I20260925 15:19:15.178258 13032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:15.191207 13032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:15.194758 13032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:15.195019 13032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:15.195840 13032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:15.197069 13032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:15.197158 13032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:15.197295 13032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:15.197366 13032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:15.228339 13032 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:43973
I20260925 15:19:15.228372 13159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:43973 every 8 connection(s)
I20260925 15:19:15.229570 13032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:15.239570 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13032
I20260925 15:19:15.239773 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:15.240182 9097 external_mini_cluster.cc:1428] Setting key 3360cedd642141ae7d3661bcac8098a8
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:15.248823 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.231984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:37101 (local address 127.8.226.126:42037)
0925 15:19:15.232152 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:15.232156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:15.233397 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 15:19:15.235096 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:15.235102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:15.235191 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:15.235299 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:15.236667 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.237426 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.238890 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.239448 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.239580 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.243480 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:15.243485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:15.243488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:15.243517 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:15.246034 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.246684 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.246691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.246694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.246844 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.247261 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.247266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.247269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.247522 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:15.247667 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:15.248278 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 15:19:15.248478 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:19:15.249707 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.232475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:42037 (local address 127.8.226.67:37101)
0925 15:19:15.233248 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:15.233289 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:15.234850 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:15.235517 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:15.235525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:15.235986 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:15.236509 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:15.236524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.237576 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.237580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:15.238752 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:15.238764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.239692 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.239701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:15.239970 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.240711 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:15.240740 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:15.243295 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:15.246164 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:15.246173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:15.246190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:15.246571 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:15.246968 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:15.246973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:15.246977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:15.247144 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:15.247641 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:15.247648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:15.247947 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 15:19:15.248204 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 15:19:15.248679 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":273,"threads_started":1}
I20260925 15:19:15.250842 13160 heartbeater.cc:347] Connected to a master server at 127.8.226.126:42037
I20260925 15:19:15.251119 13160 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:15.251878 13160 heartbeater.cc:511] Master 127.8.226.126:42037 requested a full tablet report, sending...
I20260925 15:19:15.253230 12704 ts_manager.cc:194] Registered new tserver with Master: 847359ed98ef48a5b8b7a2025f8f238d (127.8.226.67:43973)
I20260925 15:19:15.254319 12704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:37101
I20260925 15:19:15.255071 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:15.299577 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.286232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.8.226.65:45927)
0925 15:19:15.286791 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:15.286798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:15.286823 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:15.286910 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:15.286917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:15.287104 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:15.287228 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:15.288425 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.289455 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.290619 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.291100 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.291185 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.294570 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:15.294580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:15.294592 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:15.294624 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:15.297318 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.297948 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.297954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.297956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.298069 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.298401 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.298404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.298406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.298705 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:15.298914 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:15.299133 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:19:15.299317 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":310,"threads_started":1}
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:15.315718 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.303474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.8.226.126:42037)
0925 15:19:15.303680 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:15.303686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:15.303889 (+ 203us) server_negotiation.cc:407] Connection header received
0925 15:19:15.304047 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:15.304058 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:15.304113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:15.304202 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:15.305425 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.306255 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.307390 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.307779 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.307906 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.311332 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:15.311337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:15.311340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:15.311368 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:15.313470 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.314018 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.314025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.314033 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.314144 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.314463 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.314468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.314471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.314750 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:15.314996 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:15.315253 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:19:15.315446 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:19:15.316273 12703 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58932
I20260925 15:19:15.317224 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12765
I20260925 15:19:15.332082 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.318093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.8.226.65:45927)
0925 15:19:15.318248 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:15.318252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:15.321357 (+ 3105us) server_negotiation.cc:407] Connection header received
0925 15:19:15.321539 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:15.321544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:15.321636 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:15.321742 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:15.322864 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.323557 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.324901 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.325375 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.325495 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.327461 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:15.327467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:15.327471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:15.327507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:15.329867 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.330474 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.330481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.330484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.330604 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.330955 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.330959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.330961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.331243 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:15.331405 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:15.331632 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:19:15.331774 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:19:15.332598 12888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:15.411494 12893 heartbeater.cc:503] Master 127.8.226.126:42037 was elected leader, sending a full tablet report...
I20260925 15:19:15.411995 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12765
W20260925 15:19:15.427517 12689 connection.cc:535] server connection from 127.8.226.65:40189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:19:15.427719 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12899
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:19:15.442449 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.428559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.8.226.66:44443)
0925 15:19:15.429162 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:15.429169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:15.429213 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:19:15.429283 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:15.429290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:15.429508 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:15.429645 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:15.430882 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.431898 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.433132 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.433627 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.433717 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.437421 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:15.437431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:15.437444 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:15.437480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:15.440196 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.440756 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.440762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.440764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.440870 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.441314 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.441317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.441319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.441603 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:15.441816 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:15.442025 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 15:19:15.442195 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":320,"threads_started":1}
I20260925 15:19:15.443588 13022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:15.522249 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12899
I20260925 15:19:15.536886 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13032
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[12659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:19:15.552126 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.537888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.8.226.67:43973)
0925 15:19:15.538462 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:15.538469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:15.538496 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:19:15.538554 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:15.538560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:15.538773 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:15.538947 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:15.540210 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.541233 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.542492 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:15.543038 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:15.543166 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:15.547431 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:15.547445 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:15.547465 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:15.547496 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:15.550079 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.550633 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.550640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.550642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.550750 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:15.551036 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:15.551038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:15.551040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:15.551333 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:15.551528 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:15.551701 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:19:15.551865 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":284,"threads_started":1}
I20260925 15:19:15.553329 13155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:15.632094 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13032
I20260925 15:19:15.646924 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 12672
I20260925 15:19:15.648079 12734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:15.747308 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 12672
2026-09-25T15:19:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3586 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[13179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[13179](info): set up 2 sockets
Sep 25 15:19:15 dist-test-slave-8vw7 krb5kdc[13179](info): commencing operation
krb5kdc: starting...
W20260925 15:19:16.781172 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.009s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:16 dist-test-slave-8vw7 krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:16.955634 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:43379
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:43379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:17.258808 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:17.259104 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:17.259272 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:17.266022 13192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:17.266132 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:17.266201 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:17.266285 13192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:17.266372 13192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:17.275352 13192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:43379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:17.276463 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:17.277710 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:17.288769 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:17.288853 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:17.288877 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:17.289902 13192 server_base.cc:1034] running on GCE node
I20260925 15:19:17.290680 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:17.291894 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:17.293058 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790349557292992 us; error 50 us; skew 500 ppm
I20260925 15:19:17.293401 13192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:17 dist-test-slave-8vw7 krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349557, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:17.296131 13192 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:17.297235 13192 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:17.297355 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:17.297655 13192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:17.300453 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "c08a59238c3b4cdb8e512311fd2617b9"
format_stamp: "Formatted at 2026-09-25 15:19:17 on dist-test-slave-8vw7"
server_key: "9ab16b58709e119793926fff16a85a73"
server_key_iv: "43cf2548ba835bcecb829b69ff3bacc7"
server_key_version: "encryptionkey@0"
I20260925 15:19:17.301218 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "c08a59238c3b4cdb8e512311fd2617b9"
format_stamp: "Formatted at 2026-09-25 15:19:17 on dist-test-slave-8vw7"
server_key: "9ab16b58709e119793926fff16a85a73"
server_key_iv: "43cf2548ba835bcecb829b69ff3bacc7"
server_key_version: "encryptionkey@0"
I20260925 15:19:17.305425 13192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:19:17.308393 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:17.309556 13192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:17.309755 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "c08a59238c3b4cdb8e512311fd2617b9"
format_stamp: "Formatted at 2026-09-25 15:19:17 on dist-test-slave-8vw7"
server_key: "9ab16b58709e119793926fff16a85a73"
server_key_iv: "43cf2548ba835bcecb829b69ff3bacc7"
server_key_version: "encryptionkey@0"
I20260925 15:19:17.310015 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:17.345775 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:17.349821 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:17.350090 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:17.367236 13192 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:43379
I20260925 15:19:17.367254 13257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:43379 every 8 connection(s)
I20260925 15:19:17.368211 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:17.371001 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13192
I20260925 15:19:17.371227 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:17.371744 13258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:17.371809 9097 external_mini_cluster.cc:1428] Setting key b09b41725ab43bbdb9b845d53c827059
I20260925 15:19:17.385726 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9: Bootstrap starting.
Sep 25 15:19:17 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:17.389624 13258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:17.390866 13258 log.cc:824] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:17.393877 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9: No bootstrap required, opened a new log
I20260925 15:19:17.395025 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:17.373718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.8.226.126:43379)
0925 15:19:17.374518 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:17.374528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:17.374569 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:19:17.376142 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:17.376167 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:17.376523 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:17.376838 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:17.378689 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:17.380032 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:17.381752 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:17.382409 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:17.382543 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:17.387027 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:17.387037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:17.387054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:17.387113 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:17.390880 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:17.391657 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:17.391663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:17.391669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:17.391825 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:17.392219 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:17.392223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:17.392225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:17.392829 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:17.393135 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:17.393443 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:19:17.393814 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":380,"threads_started":1}
I20260925 15:19:17.401315 13258 raft_consensus.cc:348] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a59238c3b4cdb8e512311fd2617b9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43379 } }
I20260925 15:19:17.401598 13258 raft_consensus.cc:374] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:17.401664 13258 raft_consensus.cc:729] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08a59238c3b4cdb8e512311fd2617b9, State: Initialized, Role: FOLLOWER
I20260925 15:19:17.402103 13258 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a59238c3b4cdb8e512311fd2617b9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43379 } }
I20260925 15:19:17.402308 13258 raft_consensus.cc:388] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:17.402411 13258 raft_consensus.cc:482] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:17.402518 13258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:17.404659 13258 raft_consensus.cc:504] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a59238c3b4cdb8e512311fd2617b9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43379 } }
I20260925 15:19:17.405107 13258 leader_election.cc:302] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08a59238c3b4cdb8e512311fd2617b9; no voters:
I20260925 15:19:17.405560 13258 leader_election.cc:288] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:17.405725 13263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:17.406877 13263 raft_consensus.cc:686] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [term 1 LEADER]: Becoming Leader. State: Replica: c08a59238c3b4cdb8e512311fd2617b9, State: Running, Role: LEADER
I20260925 15:19:17.407457 13258 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:17.407401 13263 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a59238c3b4cdb8e512311fd2617b9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43379 } }
I20260925 15:19:17.409770 13265 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08a59238c3b4cdb8e512311fd2617b9. Latest consensus state: current_term: 1 leader_uuid: "c08a59238c3b4cdb8e512311fd2617b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a59238c3b4cdb8e512311fd2617b9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43379 } } }
I20260925 15:19:17.410053 13265 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:17.410334 13264 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08a59238c3b4cdb8e512311fd2617b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a59238c3b4cdb8e512311fd2617b9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43379 } } }
I20260925 15:19:17.410642 13264 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:17.412285 13270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:17.417896 13270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:17.426563 13270 catalog_manager.cc:1347] Generated new cluster ID: 8764cabd6a9e4b75a8238c3eeebcd9c8
I20260925 15:19:17.426671 13270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:17.438526 13270 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:17.440279 13270 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:17.448576 13270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9: Generated new TSK 0
I20260925 15:19:17.449528 13270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:19:17.506076 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:43379
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:17.814098 13284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:17.814349 13284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:17.814539 13284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:17.821476 13284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:17.821604 13284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:17.821790 13284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:17.831095 13284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:43379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:17.832409 13284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:17.833745 13284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:17.846205 13290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:17.846369 13289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:17.846576 13292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:17.847172 13284 server_base.cc:1034] running on GCE node
I20260925 15:19:17.847756 13284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:17.848639 13284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:17.849830 13284 hybrid_clock.cc:648] HybridClock initialized: now 1790349557849777 us; error 47 us; skew 500 ppm
I20260925 15:19:17.850143 13284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:17 dist-test-slave-8vw7 krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349557, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:17.853446 13284 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:19:17.854499 13284 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:17.854635 13284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:17.854934 13284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:17.857652 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "88dc8a6cba734ea79abf99571fe1a18e"
format_stamp: "Formatted at 2026-09-25 15:19:17 on dist-test-slave-8vw7"
server_key: "78b644dccd2c3be7592756daad659684"
server_key_iv: "00709eef0d654af55bcef902ce732ad0"
server_key_version: "encryptionkey@0"
I20260925 15:19:17.858456 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "88dc8a6cba734ea79abf99571fe1a18e"
format_stamp: "Formatted at 2026-09-25 15:19:17 on dist-test-slave-8vw7"
server_key: "78b644dccd2c3be7592756daad659684"
server_key_iv: "00709eef0d654af55bcef902ce732ad0"
server_key_version: "encryptionkey@0"
I20260925 15:19:17.863044 13284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260925 15:19:17.866612 13298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:17.867897 13284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:19:17.868079 13284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "88dc8a6cba734ea79abf99571fe1a18e"
format_stamp: "Formatted at 2026-09-25 15:19:17 on dist-test-slave-8vw7"
server_key: "78b644dccd2c3be7592756daad659684"
server_key_iv: "00709eef0d654af55bcef902ce732ad0"
server_key_version: "encryptionkey@0"
I20260925 15:19:17.868302 13284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:17.881912 13284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:17.885488 13284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:17.885738 13284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:17.886576 13284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:17.887828 13284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:17.887945 13284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:17.888074 13284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:17.888160 13284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:17.919831 13284 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:44941
I20260925 15:19:17.919889 13410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:44941 every 8 connection(s)
I20260925 15:19:17.920692 13284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:17.926182 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13284
I20260925 15:19:17.926429 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:17.927259 9097 external_mini_cluster.cc:1428] Setting key 529c6ef6e70611cd730d7cf0874fbcae
Sep 25 15:19:17 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349557, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:17.942740 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:17.923199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:46573 (local address 127.8.226.126:43379)
0925 15:19:17.923572 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:17.923576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:17.924592 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 15:19:17.926771 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:17.926775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:17.926841 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:17.926936 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:17.928783 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:17.929618 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:17.932064 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:17.932416 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:17.932502 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:17.936399 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:17.936404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:17.936407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:17.936436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:17.939696 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:17.940322 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:17.940329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:17.940335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:17.940468 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:17.940869 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:17.940875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:17.940881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:17.941287 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:17.941420 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:17.942010 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 15:19:17.942274 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":194,"threads_started":1}
I20260925 15:19:17.943315 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:17.923793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:43379 (local address 127.8.226.65:46573)
0925 15:19:17.924441 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:17.924478 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:17.926476 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:17.927215 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:17.927227 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:17.927929 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:17.928620 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:17.928646 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:17.929779 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:17.929785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:17.931928 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:17.931943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:17.932623 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:17.932631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:17.932932 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:17.933666 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:17.933705 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:17.936193 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:17.939839 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:17.939844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:17.939857 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:17.940217 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:17.940581 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:17.940584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:17.940598 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:17.940751 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:17.941402 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:17.941407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:17.941677 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 15:19:17.941890 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 15:19:17.942252 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":257,"threads_started":1}
I20260925 15:19:17.944785 13411 heartbeater.cc:347] Connected to a master server at 127.8.226.126:43379
I20260925 15:19:17.945140 13411 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:17.945926 13411 heartbeater.cc:511] Master 127.8.226.126:43379 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:19:17.948166 13223 ts_manager.cc:194] Registered new tserver with Master: 88dc8a6cba734ea79abf99571fe1a18e (127.8.226.65:44941)
I20260925 15:19:17.949891 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:46573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:19:17.972685 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:43379
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:18.280280 13417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:18.280552 13417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:18.280722 13417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:18.287842 13417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:18.287959 13417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:18.288131 13417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:18.297809 13417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:43379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:18.299090 13417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:18.300419 13417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:18.312526 13422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:18.312610 13423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:18.312794 13425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:18.313500 13417 server_base.cc:1034] running on GCE node
I20260925 15:19:18.314064 13417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:18.314965 13417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:18.316207 13417 hybrid_clock.cc:648] HybridClock initialized: now 1790349558316153 us; error 51 us; skew 500 ppm
I20260925 15:19:18.316519 13417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:18 dist-test-slave-8vw7 krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349558, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:18.319564 13417 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:19:18.320628 13417 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:18.320732 13417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:18.321094 13417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:18.323974 13417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "908e9134c6694964ac05f14d1421f7b3"
format_stamp: "Formatted at 2026-09-25 15:19:18 on dist-test-slave-8vw7"
server_key: "0774282a04da85488dc0bd6e9ce8a007"
server_key_iv: "502044c624289d464c34e22f4dc284cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:18.324744 13417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "908e9134c6694964ac05f14d1421f7b3"
format_stamp: "Formatted at 2026-09-25 15:19:18 on dist-test-slave-8vw7"
server_key: "0774282a04da85488dc0bd6e9ce8a007"
server_key_iv: "502044c624289d464c34e22f4dc284cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:18.329459 13417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:19:18.332577 13431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:18.333827 13417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:18.333984 13417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "908e9134c6694964ac05f14d1421f7b3"
format_stamp: "Formatted at 2026-09-25 15:19:18 on dist-test-slave-8vw7"
server_key: "0774282a04da85488dc0bd6e9ce8a007"
server_key_iv: "502044c624289d464c34e22f4dc284cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:18.334183 13417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:18.347726 13417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:18.351250 13417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:18.351495 13417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:18.352339 13417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:18.353713 13417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:18.353832 13417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:18.353950 13417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:18.354033 13417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:18.402026 13417 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:32801
I20260925 15:19:18.402069 13543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:32801 every 8 connection(s)
I20260925 15:19:18.402951 13417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:18.409122 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13417
I20260925 15:19:18.409325 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:18.409782 9097 external_mini_cluster.cc:1428] Setting key 2d5e02002ef0af62a7ea9744b6c28a2d
Sep 25 15:19:18 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349558, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:18.427904 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.406081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:36485 (local address 127.8.226.126:43379)
0925 15:19:18.406295 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:18.406301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:18.407616 (+ 1315us) server_negotiation.cc:407] Connection header received
0925 15:19:18.410734 (+ 3118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:18.410740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:18.410840 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:18.410956 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:18.413305 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.414528 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.417081 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.417812 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.417925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.422125 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:18.422134 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:18.422138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:18.422168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:18.424743 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:18.425572 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:18.425578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:18.425581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:18.425715 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:18.426126 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:18.426131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:18.426134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:18.426391 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:18.426548 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:18.427436 (+ 888us) server_negotiation.cc:299] Negotiation successful
0925 15:19:18.427636 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:19:18.429530 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.406572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:43379 (local address 127.8.226.66:36485)
0925 15:19:18.407403 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.407471 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:18.410409 (+ 2938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:18.411366 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:18.411378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:18.412164 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:18.413053 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.413083 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.414727 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.414734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.416887 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.416903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.418277 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.418287 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.418602 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.419396 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:18.419422 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:18.421890 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:18.424878 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:18.424886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:18.424902 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:18.425449 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:18.425827 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:18.425832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:18.425836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:18.426014 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:18.426511 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:18.426520 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:18.426973 (+ 453us) client_negotiation.cc:769] Sending connection context
0925 15:19:18.427339 (+ 366us) client_negotiation.cc:240] Negotiation successful
0925 15:19:18.428021 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":290,"threads_started":1}
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:19:18.431072 13544 heartbeater.cc:347] Connected to a master server at 127.8.226.126:43379
I20260925 15:19:18.431363 13544 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:18.432124 13544 heartbeater.cc:511] Master 127.8.226.126:43379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
I20260925 15:19:18.433714 13223 ts_manager.cc:194] Registered new tserver with Master: 908e9134c6694964ac05f14d1421f7b3 (127.8.226.66:32801)
I20260925 15:19:18.434962 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:36485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:19:18.456292 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:43379
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:18.758998 13549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:18.759306 13549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:18.759483 13549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:18.766400 13549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:18.766530 13549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:18.766721 13549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:18.775868 13549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:43379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:18.777035 13549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:18.778192 13549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:18.790191 13555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:18.790308 13554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:18.790211 13557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:18.790951 13549 server_base.cc:1034] running on GCE node
I20260925 15:19:18.791518 13549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:18.792338 13549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:18.793515 13549 hybrid_clock.cc:648] HybridClock initialized: now 1790349558793464 us; error 46 us; skew 500 ppm
I20260925 15:19:18.793812 13549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:18 dist-test-slave-8vw7 krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349558, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:18.803951 13549 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:19:18.805215 13549 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:18.805342 13549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:18.805639 13549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:18.808555 13549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "5911d0d7dc764078931f5c44bb2fd024"
format_stamp: "Formatted at 2026-09-25 15:19:18 on dist-test-slave-8vw7"
server_key: "5c465d144f237384de42a6b7c5180613"
server_key_iv: "090fe565c6139a233d469554329d5964"
server_key_version: "encryptionkey@0"
I20260925 15:19:18.809405 13549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "5911d0d7dc764078931f5c44bb2fd024"
format_stamp: "Formatted at 2026-09-25 15:19:18 on dist-test-slave-8vw7"
server_key: "5c465d144f237384de42a6b7c5180613"
server_key_iv: "090fe565c6139a233d469554329d5964"
server_key_version: "encryptionkey@0"
I20260925 15:19:18.814229 13549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:19:18.817142 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:18.818279 13549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:19:18.818431 13549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "5911d0d7dc764078931f5c44bb2fd024"
format_stamp: "Formatted at 2026-09-25 15:19:18 on dist-test-slave-8vw7"
server_key: "5c465d144f237384de42a6b7c5180613"
server_key_iv: "090fe565c6139a233d469554329d5964"
server_key_version: "encryptionkey@0"
I20260925 15:19:18.818673 13549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:18.833370 13549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:18.837436 13549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:18.837713 13549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:18.838624 13549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:18.839861 13549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:18.839942 13549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:18.840018 13549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:18.840070 13549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:18.871930 13549 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:46299
I20260925 15:19:18.871973 13675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:46299 every 8 connection(s)
I20260925 15:19:18.872756 13549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:18.882534 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13549
I20260925 15:19:18.882786 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:18.883209 9097 external_mini_cluster.cc:1428] Setting key 766c773e650959aef4688c9def322c39
Sep 25 15:19:18 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349558, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:18.891610 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.875132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:36459 (local address 127.8.226.126:43379)
0925 15:19:18.875308 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:18.875312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:18.876382 (+ 1070us) server_negotiation.cc:407] Connection header received
0925 15:19:18.878157 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:18.878164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:18.878260 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:18.878375 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:18.879835 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.880507 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.882091 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.882562 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.882767 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.886649 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:18.886653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:18.886657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:18.886692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:18.889181 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:18.889846 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:18.889849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:18.889851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:18.889955 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:18.890295 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:18.890298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:18.890300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:18.890532 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:18.890635 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:18.891185 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 15:19:18.891336 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:19:18.892477 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.875587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:43379 (local address 127.8.226.67:36459)
0925 15:19:18.876229 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.876267 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:18.877903 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:18.878603 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:18.878611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:18.879107 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:18.879667 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.879685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.880654 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.880659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.881944 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.881956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.882856 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.882866 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.883087 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.883823 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:18.883852 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:18.886470 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:18.889298 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:18.889304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:18.889316 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:18.889738 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:18.890042 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:18.890045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:18.890048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:18.890202 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:18.890630 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:18.890637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:18.890909 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 15:19:18.891102 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 15:19:18.891470 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260925 15:19:18.893728 13676 heartbeater.cc:347] Connected to a master server at 127.8.226.126:43379
I20260925 15:19:18.893994 13676 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:18.894577 13676 heartbeater.cc:511] Master 127.8.226.126:43379 requested a full tablet report, sending...
I20260925 15:19:18.895803 13223 ts_manager.cc:194] Registered new tserver with Master: 5911d0d7dc764078931f5c44bb2fd024 (127.8.226.67:46299)
I20260925 15:19:18.896597 13223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:36459
I20260925 15:19:18.898412 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:18.914387 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.902784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.8.226.126:43379)
0925 15:19:18.902923 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:18.902927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:18.903271 (+ 344us) server_negotiation.cc:407] Connection header received
0925 15:19:18.903388 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:18.903391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:18.903444 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:18.903519 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:18.904813 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.905566 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.906729 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.907142 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.907227 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.909591 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:18.909604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:18.909607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:18.909634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:18.911983 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:18.912568 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:18.912571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:18.912573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:18.912685 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:18.913083 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:18.913086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:18.913088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:18.913465 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:18.913654 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:18.913949 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:19:18.914131 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:19:18.917923 13223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:18.925055 13223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:18.947605 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.938151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.8.226.65:44941)
0925 15:19:18.938899 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:18.938910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:18.938945 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:19:18.939057 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:18.939067 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:18.939350 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:18.940253 (+ 903us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:18.940486 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.942498 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.945336 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.946530 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.946657 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.946853 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:18.947136 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:19:18.947291 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":517,"threads_started":1}
I20260925 15:19:18.947867 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.936623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:46299 (local address 127.0.0.1:47232)
0925 15:19:18.937456 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.937498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:18.937655 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:18.938535 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:18.938543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:18.938573 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:18.938959 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.938977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.941869 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.941877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.945433 (+ 3556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.945441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.946996 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.947001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.947160 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.947188 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:19:18.947271 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:19:18.947414 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":707,"thread_start_us":862,"threads_started":1}
I20260925 15:19:18.948592 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.936782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.8.226.67:46299)
0925 15:19:18.937799 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:18.937807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:18.937840 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:18.937935 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:18.937944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:18.938270 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:18.939210 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:18.939557 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.941682 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.945565 (+ 3883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.946638 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.946766 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.946967 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:18.948075 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0925 15:19:18.948256 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":1718,"threads_started":1}
I20260925 15:19:18.948751 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.937582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:44941 (local address 127.0.0.1:37516)
0925 15:19:18.938214 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.938230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:18.938390 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:18.939575 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:18.939579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:18.939604 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:18.940068 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.940080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.942725 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.942733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.945194 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.945207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.946737 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.946743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.946907 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.946962 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:19:18.947091 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:19:18.948374 (+ 1283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":236,"thread_start_us":768,"threads_started":1}
I20260925 15:19:18.951861 13611 tablet_service.cc:1467] Processing CreateTablet for tablet c211a667b9184c9e8951c8ef35640c5b (DEFAULT_TABLE table=test-table [id=20c70ffbe3ad4c3d895c21041e1a198b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:18.952020 13346 tablet_service.cc:1467] Processing CreateTablet for tablet c211a667b9184c9e8951c8ef35640c5b (DEFAULT_TABLE table=test-table [id=20c70ffbe3ad4c3d895c21041e1a198b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:18.953168 13411 heartbeater.cc:503] Master 127.8.226.126:43379 was elected leader, sending a full tablet report...
I20260925 15:19:18.953317 13611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c211a667b9184c9e8951c8ef35640c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:18.953374 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.938211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:32801 (local address 127.0.0.1:47692)
0925 15:19:18.944796 (+ 6585us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.944818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:18.944918 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:18.945591 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:18.945595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:18.945619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:18.945927 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.945937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.947532 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.947542 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.950684 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:18.950698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.952640 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.952648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:18.952820 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.952850 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:19:18.952943 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:19:18.953094 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"thread_start_us":6318,"threads_started":1}
I20260925 15:19:18.953382 13346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c211a667b9184c9e8951c8ef35640c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:18.954089 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.938696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.8.226.66:32801)
0925 15:19:18.945081 (+ 6385us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:18.945089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:18.945118 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:19:18.945192 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:18.945197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:18.945408 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:18.945568 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:18.946094 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.947386 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.950863 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:18.952205 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:18.952336 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:18.952500 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:18.953611 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0925 15:19:18.953767 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6220,"thread_start_us":6099,"threads_started":1}
I20260925 15:19:18.957720 13479 tablet_service.cc:1467] Processing CreateTablet for tablet c211a667b9184c9e8951c8ef35640c5b (DEFAULT_TABLE table=test-table [id=20c70ffbe3ad4c3d895c21041e1a198b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:18.958963 13479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c211a667b9184c9e8951c8ef35640c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:18.966777 13694 tablet_bootstrap.cc:492] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Bootstrap starting.
I20260925 15:19:18.967880 13695 tablet_bootstrap.cc:492] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Bootstrap starting.
I20260925 15:19:18.971503 13694 tablet_bootstrap.cc:654] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:18.971905 13695 tablet_bootstrap.cc:654] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:18.971942 13696 tablet_bootstrap.cc:492] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Bootstrap starting.
I20260925 15:19:18.973091 13694 log.cc:824] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:18.973130 13695 log.cc:824] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:18.975150 13696 tablet_bootstrap.cc:654] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:18.976032 13696 log.cc:824] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:18.976532 13694 tablet_bootstrap.cc:492] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: No bootstrap required, opened a new log
I20260925 15:19:18.976532 13695 tablet_bootstrap.cc:492] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: No bootstrap required, opened a new log
I20260925 15:19:18.976785 13695 ts_tablet_manager.cc:1397] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 15:19:18.976811 13694 ts_tablet_manager.cc:1397] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 15:19:18.978209 13696 tablet_bootstrap.cc:492] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: No bootstrap required, opened a new log
I20260925 15:19:18.978473 13696 ts_tablet_manager.cc:1397] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:19:18.984684 13696 raft_consensus.cc:348] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:18.985110 13696 raft_consensus.cc:374] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:18.985229 13696 raft_consensus.cc:729] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908e9134c6694964ac05f14d1421f7b3, State: Initialized, Role: FOLLOWER
I20260925 15:19:18.985884 13696 consensus_queue.cc:260] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:18.986932 13544 heartbeater.cc:503] Master 127.8.226.126:43379 was elected leader, sending a full tablet report...
I20260925 15:19:18.986712 13694 raft_consensus.cc:348] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:18.987076 13694 raft_consensus.cc:374] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:18.987156 13694 raft_consensus.cc:729] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dc8a6cba734ea79abf99571fe1a18e, State: Initialized, Role: FOLLOWER
I20260925 15:19:18.987264 13695 raft_consensus.cc:348] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:18.987541 13696 ts_tablet_manager.cc:1428] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260925 15:19:18.987600 13695 raft_consensus.cc:374] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:18.987689 13695 raft_consensus.cc:729] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5911d0d7dc764078931f5c44bb2fd024, State: Initialized, Role: FOLLOWER
I20260925 15:19:18.987789 13694 consensus_queue.cc:260] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:18.988188 13695 consensus_queue.cc:260] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:18.989430 13694 ts_tablet_manager.cc:1428] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:19:18.989881 13695 ts_tablet_manager.cc:1428] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 15:19:18.989905 13676 heartbeater.cc:503] Master 127.8.226.126:43379 was elected leader, sending a full tablet report...
I20260925 15:19:19.025684 13702 raft_consensus.cc:482] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:19.026005 13702 raft_consensus.cc:504] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:19.027568 13702 leader_election.cc:288] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 908e9134c6694964ac05f14d1421f7b3 (127.8.226.66:32801), 88dc8a6cba734ea79abf99571fe1a18e (127.8.226.65:44941)
I20260925 15:19:19.033151 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.027807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:44941 (local address 127.8.226.67:39161)
0925 15:19:19.028103 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:19.028125 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:19.028255 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:19.028696 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:19.028699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:19.028721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:19.028887 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:19.028894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.029934 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.029942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:19.031256 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:19.031267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.032342 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.032348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:19.032528 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.032598 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 15:19:19.032692 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:19:19.032848 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 15:19:19.033181 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.028045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:39161 (local address 127.8.226.65:44941)
0925 15:19:19.028202 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:19.028208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:19.028227 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:19:19.028447 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:19.028453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:19.028563 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:19.028671 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:19.029061 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.029799 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.031431 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.032185 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.032363 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.032439 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:19.032741 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 15:19:19.032892 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:19:19.034214 13366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c211a667b9184c9e8951c8ef35640c5b" candidate_uuid: "5911d0d7dc764078931f5c44bb2fd024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dc8a6cba734ea79abf99571fe1a18e" is_pre_election: true
I20260925 15:19:19.034621 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.028001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:57303 (local address 127.8.226.66:32801)
0925 15:19:19.028185 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:19.028201 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:19.028919 (+ 718us) server_negotiation.cc:407] Connection header received
0925 15:19:19.029134 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:19.029137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:19.029196 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:19.029290 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:19.029793 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.030486 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.032184 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.033685 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.033883 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.033950 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:19.034267 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:19:19.034385 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:19:19.034685 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.027919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:32801 (local address 127.8.226.67:57303)
0925 15:19:19.028794 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:19.028812 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:19.028959 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:19.029302 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:19.029306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:19.029326 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:19.029640 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:19.029649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.030632 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.030639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:19.032028 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:19.032039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.033891 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.033897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:19.034098 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.034129 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:19:19.034244 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:19:19.034368 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":493,"threads_started":1}
I20260925 15:19:19.034719 13366 raft_consensus.cc:2454] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5911d0d7dc764078931f5c44bb2fd024 in term 0.
I20260925 15:19:19.035322 13565 leader_election.cc:302] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5911d0d7dc764078931f5c44bb2fd024, 88dc8a6cba734ea79abf99571fe1a18e; no voters:
I20260925 15:19:19.035840 13702 raft_consensus.cc:2781] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:19.035764 13499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c211a667b9184c9e8951c8ef35640c5b" candidate_uuid: "5911d0d7dc764078931f5c44bb2fd024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908e9134c6694964ac05f14d1421f7b3" is_pre_election: true
I20260925 15:19:19.036017 13702 raft_consensus.cc:482] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:19.036120 13702 raft_consensus.cc:3037] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:19.036279 13499 raft_consensus.cc:2454] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5911d0d7dc764078931f5c44bb2fd024 in term 0.
I20260925 15:19:19.038990 13702 raft_consensus.cc:504] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:19.039619 13702 leader_election.cc:288] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [CANDIDATE]: Term 1 election: Requested vote from peers 908e9134c6694964ac05f14d1421f7b3 (127.8.226.66:32801), 88dc8a6cba734ea79abf99571fe1a18e (127.8.226.65:44941)
I20260925 15:19:19.039954 13499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c211a667b9184c9e8951c8ef35640c5b" candidate_uuid: "5911d0d7dc764078931f5c44bb2fd024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908e9134c6694964ac05f14d1421f7b3"
I20260925 15:19:19.040184 13499 raft_consensus.cc:3037] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:19.040366 13366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c211a667b9184c9e8951c8ef35640c5b" candidate_uuid: "5911d0d7dc764078931f5c44bb2fd024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dc8a6cba734ea79abf99571fe1a18e"
I20260925 15:19:19.040563 13366 raft_consensus.cc:3037] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:19.043380 13499 raft_consensus.cc:2454] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5911d0d7dc764078931f5c44bb2fd024 in term 1.
I20260925 15:19:19.043763 13366 raft_consensus.cc:2454] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5911d0d7dc764078931f5c44bb2fd024 in term 1.
I20260925 15:19:19.043733 13567 leader_election.cc:302] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5911d0d7dc764078931f5c44bb2fd024, 908e9134c6694964ac05f14d1421f7b3; no voters:
I20260925 15:19:19.044003 13702 raft_consensus.cc:2781] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:19.044447 13702 raft_consensus.cc:686] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 1 LEADER]: Becoming Leader. State: Replica: 5911d0d7dc764078931f5c44bb2fd024, State: Running, Role: LEADER
I20260925 15:19:19.044942 13702 consensus_queue.cc:237] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } }
I20260925 15:19:19.051595 13223 catalog_manager.cc:5495] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5911d0d7dc764078931f5c44bb2fd024 (127.8.226.67). New cstate: current_term: 1 leader_uuid: "5911d0d7dc764078931f5c44bb2fd024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5911d0d7dc764078931f5c44bb2fd024" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:19.067936 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.063404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.8.226.67:46299)
0925 15:19:19.063569 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:19.063573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:19.063704 (+ 131us) server_negotiation.cc:407] Connection header received
0925 15:19:19.063861 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:19.063864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:19.063932 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:19.064040 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:19.064307 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.065020 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.066235 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.066624 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.066710 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.067090 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:19.067543 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:19.067644 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:19.067704 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:19:19.067763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:19:19.075419 13366 raft_consensus.cc:1261] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 1 FOLLOWER]: Refusing update from remote peer 5911d0d7dc764078931f5c44bb2fd024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:19.075413 13499 raft_consensus.cc:1261] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 1 FOLLOWER]: Refusing update from remote peer 5911d0d7dc764078931f5c44bb2fd024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:19.076318 13704 consensus_queue.cc:1035] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "908e9134c6694964ac05f14d1421f7b3" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:19.076658 13702 consensus_queue.cc:1035] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88dc8a6cba734ea79abf99571fe1a18e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:19.086880 13707 mvcc.cc:205] Tried to move back new op lower bound from 7333271793963839488 to 7333271793853956096. Current Snapshot: MvccSnapshot[applied={T|T < 7333271793963839488}]
I20260925 15:19:19.089438 13709 mvcc.cc:205] Tried to move back new op lower bound from 7333271793963839488 to 7333271793853956096. Current Snapshot: MvccSnapshot[applied={T|T < 7333271793963839488}]
I20260925 15:19:19.101059 13223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:19.101318 13223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:19.104816 13223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c08a59238c3b4cdb8e512311fd2617b9: Sending DeleteTablet for 3 replicas of tablet c211a667b9184c9e8951c8ef35640c5b
I20260925 15:19:19.106241 13479 tablet_service.cc:1514] Processing DeleteTablet for tablet c211a667b9184c9e8951c8ef35640c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:19 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47692
I20260925 15:19:19.106247 13346 tablet_service.cc:1514] Processing DeleteTablet for tablet c211a667b9184c9e8951c8ef35640c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:19 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:37516
I20260925 15:19:19.106278 13611 tablet_service.cc:1514] Processing DeleteTablet for tablet c211a667b9184c9e8951c8ef35640c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:19 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47232
I20260925 15:19:19.107890 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13284
I20260925 15:19:19.141847 13717 tablet_replica.cc:330] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: stopping tablet replica
I20260925 15:19:19.142373 13717 raft_consensus.cc:2229] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:19.143219 13717 raft_consensus.cc:2258] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:19.143864 13718 tablet_replica.cc:330] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: stopping tablet replica
I20260925 15:19:19.144189 13718 raft_consensus.cc:2229] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:19.144608 13718 raft_consensus.cc:2258] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:19.147514 13718 ts_tablet_manager.cc:1905] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:19.150661 13719 tablet_replica.cc:330] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: stopping tablet replica
Sep 25 15:19:19 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:19.151063 13719 raft_consensus.cc:2229] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:19.151484 13719 raft_consensus.cc:2258] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:19.151543 13717 ts_tablet_manager.cc:1905] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:19.153221 13719 ts_tablet_manager.cc:1905] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:19.155855 13718 ts_tablet_manager.cc:1918] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:19.156018 13718 log.cc:1196] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/c211a667b9184c9e8951c8ef35640c5b
I20260925 15:19:19.156504 13718 ts_tablet_manager.cc:1939] T c211a667b9184c9e8951c8ef35640c5b P 88dc8a6cba734ea79abf99571fe1a18e: Deleting consensus metadata
I20260925 15:19:19.158102 13717 ts_tablet_manager.cc:1918] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:19.158242 13717 log.cc:1196] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/c211a667b9184c9e8951c8ef35640c5b
I20260925 15:19:19.158264 13208 catalog_manager.cc:4841] TS 88dc8a6cba734ea79abf99571fe1a18e (127.8.226.65:44941): tablet c211a667b9184c9e8951c8ef35640c5b (table test-table [id=20c70ffbe3ad4c3d895c21041e1a198b]) successfully deleted
I20260925 15:19:19.158763 13717 ts_tablet_manager.cc:1939] T c211a667b9184c9e8951c8ef35640c5b P 5911d0d7dc764078931f5c44bb2fd024: Deleting consensus metadata
I20260925 15:19:19.160233 13719 ts_tablet_manager.cc:1918] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:19.160367 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.109129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.8.226.65:44941)
0925 15:19:19.109344 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:19.109353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:19.133582 (+ 24229us) server_negotiation.cc:407] Connection header received
0925 15:19:19.134123 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:19.134132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:19.134239 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:19.134325 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:19.136498 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.138125 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.145095 (+ 6970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.145492 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.145561 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.153212 (+ 7651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:19.153224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:19.153250 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:19.153289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:19.157104 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:19.157872 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:19.157879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:19.157881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:19.158032 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:19.158487 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:19.158491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:19.158494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:19.158902 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:19.159212 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:19.159780 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 15:19:19.159996 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:19:19.160372 13210 catalog_manager.cc:4841] TS 5911d0d7dc764078931f5c44bb2fd024 (127.8.226.67:46299): tablet c211a667b9184c9e8951c8ef35640c5b (table test-table [id=20c70ffbe3ad4c3d895c21041e1a198b]) successfully deleted
I20260925 15:19:19.160661 13719 log.cc:1196] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/c211a667b9184c9e8951c8ef35640c5b
I20260925 15:19:19.161197 13719 ts_tablet_manager.cc:1939] T c211a667b9184c9e8951c8ef35640c5b P 908e9134c6694964ac05f14d1421f7b3: Deleting consensus metadata
I20260925 15:19:19.161880 13406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:19.162982 13210 catalog_manager.cc:4841] TS 908e9134c6694964ac05f14d1421f7b3 (127.8.226.66:32801): tablet c211a667b9184c9e8951c8ef35640c5b (table test-table [id=20c70ffbe3ad4c3d895c21041e1a198b]) successfully deleted
I20260925 15:19:19.254812 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13284
I20260925 15:19:19.271601 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13417
Sep 25 15:19:19 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:19:19.287556 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.272451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.8.226.66:32801)
0925 15:19:19.272626 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:19.272631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:19.272772 (+ 141us) server_negotiation.cc:407] Connection header received
0925 15:19:19.272988 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:19.272997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:19.273074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:19.273189 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:19.274737 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.275429 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.277284 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.277657 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.277779 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.281998 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:19.282011 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:19.282025 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:19.282056 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:19.284734 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:19.285554 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:19.285561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:19.285563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:19.285679 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:19.286064 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:19.286073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:19.286075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:19.286423 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:19.286720 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:19.287017 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 15:19:19.287246 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:19:19.288417 13539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:19.378710 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13417
I20260925 15:19:19.400857 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13549
Sep 25 15:19:19 dist-test-slave-8vw7 krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:19:19.418648 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.401657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.8.226.67:46299)
0925 15:19:19.401851 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:19.401856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:19.401876 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:19:19.401968 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:19.401973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:19.402063 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:19.402172 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:19.403389 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.404601 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.405897 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:19.406646 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:19.406769 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:19.411260 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:19.411280 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:19.411300 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:19.411337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:19.415884 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:19.416480 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:19.416490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:19.416494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:19.416671 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:19.417025 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:19.417030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:19.417033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:19.417494 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:19.417838 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:19.418009 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:19:19.418233 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:19:19.419396 13671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:19.538990 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13549
I20260925 15:19:19.553184 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13192
I20260925 15:19:19.554265 13253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:19.661939 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13192
2026-09-25T15:19:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3916 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:19Z Disabled control of system clock
I20260925 15:19:19.725250 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35501
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:35501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:19:20.022461 13728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:20.022698 13728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:20.022773 13728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:20.029199 13728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:20.029291 13728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:20.029328 13728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:20.029384 13728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:20.029426 13728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:20.037583 13728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:35501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:20.038704 13728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:20.040319 13728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:20.050854 13733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:20.050843 13734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:20.050886 13736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:20.051623 13728 server_base.cc:1034] running on GCE node
I20260925 15:19:20.052273 13728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:20.053561 13728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:20.054797 13728 hybrid_clock.cc:648] HybridClock initialized: now 1790349560054766 us; error 52 us; skew 500 ppm
I20260925 15:19:20.055123 13728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:20.057309 13728 webserver.cc:460] Webserver started at http://127.8.226.126:37087/ using document root <none> and password file <none>
I20260925 15:19:20.057904 13728 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:20.058048 13728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:20.058354 13728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:20.061055 13728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "181c628d4da248ddbd3aac9cb201618d"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "692bc0575c05d803a5d5a286ad70a7fa"
server_key_iv: "308891e2679266812b1b0dbaf85df532"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.061813 13728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "181c628d4da248ddbd3aac9cb201618d"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "692bc0575c05d803a5d5a286ad70a7fa"
server_key_iv: "308891e2679266812b1b0dbaf85df532"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.066334 13728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:19:20.069661 13742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:20.070799 13728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:19:20.070984 13728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "181c628d4da248ddbd3aac9cb201618d"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "692bc0575c05d803a5d5a286ad70a7fa"
server_key_iv: "308891e2679266812b1b0dbaf85df532"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.071199 13728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:20.076368 13728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:20.077147 13728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:20.077399 13728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:20.094103 13728 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:35501
I20260925 15:19:20.094139 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:35501 every 8 connection(s)
I20260925 15:19:20.095222 13728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:20.098428 13794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:20.099751 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13728
I20260925 15:19:20.099947 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:20.100514 9097 external_mini_cluster.cc:1428] Setting key 4301ea7d762ff2298fff88ac875a8dd0
I20260925 15:19:20.108448 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:20.102382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.8.226.126:35501)
0925 15:19:20.103415 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:20.103424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:20.103464 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:19:20.105019 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:20.105039 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:20.105436 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:20.105714 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:20.106759 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:20.106764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:20.106775 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:20.106791 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:20.106851 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:20.106865 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:20.107022 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:20.107155 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:19:20.107437 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":371,"threads_started":1}
I20260925 15:19:20.109782 13794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d: Bootstrap starting.
I20260925 15:19:20.112838 13794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:20.113888 13794 log.cc:824] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:20.116542 13794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d: No bootstrap required, opened a new log
I20260925 15:19:20.123899 13794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c628d4da248ddbd3aac9cb201618d" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35501 } }
I20260925 15:19:20.124198 13794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:20.124262 13794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181c628d4da248ddbd3aac9cb201618d, State: Initialized, Role: FOLLOWER
I20260925 15:19:20.124699 13794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c628d4da248ddbd3aac9cb201618d" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35501 } }
I20260925 15:19:20.124882 13794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:20.125026 13794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:20.125197 13794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:20.127209 13794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c628d4da248ddbd3aac9cb201618d" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35501 } }
I20260925 15:19:20.127660 13794 leader_election.cc:302] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181c628d4da248ddbd3aac9cb201618d; no voters:
I20260925 15:19:20.128222 13794 leader_election.cc:288] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:20.128324 13799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:20.129106 13799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [term 1 LEADER]: Becoming Leader. State: Replica: 181c628d4da248ddbd3aac9cb201618d, State: Running, Role: LEADER
I20260925 15:19:20.129659 13799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c628d4da248ddbd3aac9cb201618d" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35501 } }
I20260925 15:19:20.130175 13794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:20.132159 13800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181c628d4da248ddbd3aac9cb201618d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c628d4da248ddbd3aac9cb201618d" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35501 } } }
I20260925 15:19:20.132483 13800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:20.133392 13805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:20.133221 13801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181c628d4da248ddbd3aac9cb201618d. Latest consensus state: current_term: 1 leader_uuid: "181c628d4da248ddbd3aac9cb201618d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c628d4da248ddbd3aac9cb201618d" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35501 } } }
I20260925 15:19:20.133613 13801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:20.138828 13805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:20.149346 13805 catalog_manager.cc:1347] Generated new cluster ID: 0329bdd8af3548eda508e7d4e0e7db75
I20260925 15:19:20.149473 13805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:20.168781 13805 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:20.169216 13805 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:20.185308 13805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d: Generated new TSK 0
I20260925 15:19:20.186048 13805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:20.191736 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35501
--builtin_ntp_servers=127.8.226.84:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:19:20.505853 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:20.506112 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:20.506192 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:20.513224 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:20.513324 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:20.513427 13818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:20.522063 13818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:20.523204 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:20.524784 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:20.538185 13823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:20.538179 13826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:20.538130 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:20.538954 13818 server_base.cc:1034] running on GCE node
I20260925 15:19:20.539510 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:20.540335 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:20.541512 13818 hybrid_clock.cc:648] HybridClock initialized: now 1790349560541456 us; error 46 us; skew 500 ppm
I20260925 15:19:20.541847 13818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:20.543992 13818 webserver.cc:460] Webserver started at http://127.8.226.65:43209/ using document root <none> and password file <none>
I20260925 15:19:20.544662 13818 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:20.544775 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:20.545135 13818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:20.547832 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "293c82010c414b9b8afd9190d25f3876"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "9bc840d907d7038d06b7e05b0888ef9d"
server_key_iv: "7418adb62bdccc79808f05bc2e21b262"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.548614 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "293c82010c414b9b8afd9190d25f3876"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "9bc840d907d7038d06b7e05b0888ef9d"
server_key_iv: "7418adb62bdccc79808f05bc2e21b262"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.553781 13818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:19:20.557178 13832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:20.558506 13818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:19:20.558676 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "293c82010c414b9b8afd9190d25f3876"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "9bc840d907d7038d06b7e05b0888ef9d"
server_key_iv: "7418adb62bdccc79808f05bc2e21b262"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.558912 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:20.564640 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:20.565426 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:20.565653 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:20.566468 13818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:20.567823 13818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:20.567953 13818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:20.568078 13818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:20.568176 13818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:20.598829 13818 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:46407
I20260925 15:19:20.598912 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:46407 every 8 connection(s)
I20260925 15:19:20.600136 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:20.602352 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13818
I20260925 15:19:20.602540 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:20.603223 9097 external_mini_cluster.cc:1428] Setting key b1e26af32dfd29a72c9dca7122a2c5b7
I20260925 15:19:20.606281 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35501
--builtin_ntp_servers=127.8.226.84:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:19:20.609155 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:20.602633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:37315 (local address 127.8.226.126:35501)
0925 15:19:20.602837 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:20.602843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:20.604374 (+ 1531us) server_negotiation.cc:407] Connection header received
0925 15:19:20.606531 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:20.606539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:20.606598 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:20.606665 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:20.608405 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:20.608412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:20.608419 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:20.608439 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:20.608477 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:20.608493 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:20.608600 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:20.608836 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:19:20.608892 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:19:20.610347 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:20.603138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35501 (local address 127.8.226.65:37315)
0925 15:19:20.604193 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:20.604250 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:20.606209 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:20.607168 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:20.607181 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:20.607316 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:20.608065 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:20.608071 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:20.608119 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:20.608223 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:20.608595 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:20.608626 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:20.608637 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:19:20.608899 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:19:20.609164 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":296,"threads_started":1}
I20260925 15:19:20.611728 13945 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35501
I20260925 15:19:20.611999 13945 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:20.612675 13945 heartbeater.cc:511] Master 127.8.226.126:35501 requested a full tablet report, sending...
I20260925 15:19:20.614621 13759 ts_manager.cc:194] Registered new tserver with Master: 293c82010c414b9b8afd9190d25f3876 (127.8.226.65:46407)
W20260925 15:19:20.910218 13948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:20.910462 13948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:20.910609 13948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:20.917415 13948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:20.917497 13948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:20.917698 13948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:20.926265 13948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:20.927421 13948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:20.929262 13948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:20.941934 13953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:20.941897 13954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:20.941984 13956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:20.942643 13948 server_base.cc:1034] running on GCE node
I20260925 15:19:20.943166 13948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:20.943966 13948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:20.945168 13948 hybrid_clock.cc:648] HybridClock initialized: now 1790349560945122 us; error 41 us; skew 500 ppm
I20260925 15:19:20.945469 13948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:20.947430 13948 webserver.cc:460] Webserver started at http://127.8.226.66:44041/ using document root <none> and password file <none>
I20260925 15:19:20.948051 13948 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:20.948171 13948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:20.948467 13948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:20.951198 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "3ded1eee63a54277ab4d65da49150cf7"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "7f19065a7f9812e3ed22bcec87716ae7"
server_key_iv: "9602a16aa8ec4d617372c5af7e425732"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.951949 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "3ded1eee63a54277ab4d65da49150cf7"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "7f19065a7f9812e3ed22bcec87716ae7"
server_key_iv: "9602a16aa8ec4d617372c5af7e425732"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.956653 13948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260925 15:19:20.960091 13962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:20.961323 13948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:20.961486 13948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "3ded1eee63a54277ab4d65da49150cf7"
format_stamp: "Formatted at 2026-09-25 15:19:20 on dist-test-slave-8vw7"
server_key: "7f19065a7f9812e3ed22bcec87716ae7"
server_key_iv: "9602a16aa8ec4d617372c5af7e425732"
server_key_version: "encryptionkey@0"
I20260925 15:19:20.961695 13948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:20.967420 13948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:20.968120 13948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:20.968370 13948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:20.969257 13948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:20.970547 13948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:20.970659 13948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:20.970783 13948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:20.970880 13948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:21.001780 13948 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:36813
I20260925 15:19:21.001861 14074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:36813 every 8 connection(s)
I20260925 15:19:21.003089 13948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:21.005228 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 13948
I20260925 15:19:21.005374 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:21.005972 9097 external_mini_cluster.cc:1428] Setting key 55332c7055b238c9c70896c6ad5b40cd
I20260925 15:19:21.008988 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35501
--builtin_ntp_servers=127.8.226.84:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:19:21.012557 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.005732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:50851 (local address 127.8.226.126:35501)
0925 15:19:21.005956 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.005975 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.007542 (+ 1567us) server_negotiation.cc:407] Connection header received
0925 15:19:21.010152 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.010156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.010202 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.010283 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.011896 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.011901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.011905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.011923 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.011971 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.011987 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.012085 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.012302 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.012351 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:19:21.013844 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.006321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35501 (local address 127.8.226.66:50851)
0925 15:19:21.007376 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.007440 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.009858 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.010561 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.010575 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.010703 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.011504 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.011510 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.011560 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.011674 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.012090 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.012104 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.012115 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.012376 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.012625 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":373,"threads_started":1}
I20260925 15:19:21.015251 14075 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35501
I20260925 15:19:21.015550 14075 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:21.016124 14075 heartbeater.cc:511] Master 127.8.226.126:35501 requested a full tablet report, sending...
I20260925 15:19:21.017599 13759 ts_manager.cc:194] Registered new tserver with Master: 3ded1eee63a54277ab4d65da49150cf7 (127.8.226.66:36813)
W20260925 15:19:21.321676 14078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:21.321996 14078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:21.322152 14078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:21.329041 14078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:21.329161 14078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:21.329327 14078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:21.338284 14078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:21.339524 14078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:21.341231 14078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:21.353220 14083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:21.353351 14084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:21.353348 14086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:21.354065 14078 server_base.cc:1034] running on GCE node
I20260925 15:19:21.354671 14078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:21.355540 14078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:21.356750 14078 hybrid_clock.cc:648] HybridClock initialized: now 1790349561356721 us; error 46 us; skew 500 ppm
I20260925 15:19:21.357071 14078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:21.359261 14078 webserver.cc:460] Webserver started at http://127.8.226.67:35363/ using document root <none> and password file <none>
I20260925 15:19:21.359853 14078 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:21.359934 14078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:21.360191 14078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:21.363404 14078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "5db5fd8349944df59d2f1d478baef618"
format_stamp: "Formatted at 2026-09-25 15:19:21 on dist-test-slave-8vw7"
server_key: "f5af944196dd91979031c14b22be1df6"
server_key_iv: "f42a0a5765cb3d6e9115edb265b0a39f"
server_key_version: "encryptionkey@0"
I20260925 15:19:21.364070 14078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "5db5fd8349944df59d2f1d478baef618"
format_stamp: "Formatted at 2026-09-25 15:19:21 on dist-test-slave-8vw7"
server_key: "f5af944196dd91979031c14b22be1df6"
server_key_iv: "f42a0a5765cb3d6e9115edb265b0a39f"
server_key_version: "encryptionkey@0"
I20260925 15:19:21.368777 14078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:19:21.371869 14092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:21.373092 14078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:21.373252 14078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "5db5fd8349944df59d2f1d478baef618"
format_stamp: "Formatted at 2026-09-25 15:19:21 on dist-test-slave-8vw7"
server_key: "f5af944196dd91979031c14b22be1df6"
server_key_iv: "f42a0a5765cb3d6e9115edb265b0a39f"
server_key_version: "encryptionkey@0"
I20260925 15:19:21.373461 14078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:21.378662 14078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:21.379402 14078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:21.379716 14078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:21.380846 14078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:21.382149 14078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:21.382264 14078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:21.382378 14078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:21.382462 14078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:21.413558 14078 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:33037
I20260925 15:19:21.413606 14204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:33037 every 8 connection(s)
I20260925 15:19:21.414875 14078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:21.419257 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14078
I20260925 15:19:21.419433 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:21.419905 9097 external_mini_cluster.cc:1428] Setting key df85be6bbcf7bbbdba1beb61089437dc
I20260925 15:19:21.422993 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.417563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:53319 (local address 127.8.226.126:35501)
0925 15:19:21.417760 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.417764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.419108 (+ 1344us) server_negotiation.cc:407] Connection header received
0925 15:19:21.420880 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.420886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.420987 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.421110 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.422282 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.422287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.422292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.422309 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.422349 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.422361 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.422483 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.422670 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.422735 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:19:21.423846 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.418083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35501 (local address 127.8.226.67:53319)
0925 15:19:21.418951 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.418992 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.420613 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.421323 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.421331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.421418 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.422011 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.422013 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.422048 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.422126 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.422450 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.422461 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.422470 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.422764 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.423011 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":260,"threads_started":1}
I20260925 15:19:21.425042 14205 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35501
I20260925 15:19:21.425307 14205 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:21.425840 14205 heartbeater.cc:511] Master 127.8.226.126:35501 requested a full tablet report, sending...
I20260925 15:19:21.427040 13759 ts_manager.cc:194] Registered new tserver with Master: 5db5fd8349944df59d2f1d478baef618 (127.8.226.67:33037)
I20260925 15:19:21.434767 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:21.441411 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.438912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.8.226.126:35501)
0925 15:19:21.439055 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.439059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.439582 (+ 523us) server_negotiation.cc:407] Connection header received
0925 15:19:21.439764 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.439775 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.439816 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.439935 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.440799 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.440804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.440808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.440824 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.440861 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.440871 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.441000 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.441115 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.441169 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:19:21.444758 13759 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:21.451187 13759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:21.476977 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.461402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36813 (local address 127.0.0.1:42320)
0925 15:19:21.472533 (+ 11131us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.472595 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.472764 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.474978 (+ 2214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.474986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.475078 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.475827 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.475830 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.475851 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.475908 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.476287 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.476297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.476304 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.476477 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.476555 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10809,"threads_started":1}
I20260925 15:19:21.476994 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.461615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.8.226.66:36813)
0925 15:19:21.473555 (+ 11940us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.473567 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.473591 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:19:21.473645 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.473650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.473825 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.473988 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.476077 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.476086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.476107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.476122 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.476166 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.476181 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.476359 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.476465 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.476559 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11798,"thread_start_us":11685,"threads_started":1}
I20260925 15:19:21.479372 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.461651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:54990)
0925 15:19:21.477166 (+ 15515us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.477183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.477278 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.477687 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.477692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.477713 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.478236 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.478239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.478253 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.478294 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.478969 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.478974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.478978 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.479057 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.479103 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15423,"thread_start_us":10405,"threads_started":1}
I20260925 15:19:21.479491 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.461914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.8.226.65:46407)
0925 15:19:21.473277 (+ 11363us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.473286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.477281 (+ 3995us) server_negotiation.cc:407] Connection header received
0925 15:19:21.477380 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.477387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.477554 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.477752 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.478793 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.478799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.478808 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.478823 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.478857 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.478869 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.479010 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.479085 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.479169 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11206,"thread_start_us":11083,"threads_started":1}
I20260925 15:19:21.483289 13880 tablet_service.cc:1467] Processing CreateTablet for tablet b6d3e6bbfafd4cd5bed36378037c203f (DEFAULT_TABLE table=test-table [id=f7654b0eeefe4db98db7417c426c8dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:21.484920 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.462887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:33037 (local address 127.0.0.1:46434)
0925 15:19:21.479828 (+ 16941us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.479844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.479923 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.483582 (+ 3659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.483587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.483607 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.484115 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.484117 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.484129 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.484160 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.484518 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.484522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.484527 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.484619 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.484664 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16856,"thread_start_us":20031,"threads_started":1}
I20260925 15:19:21.485134 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.463538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.8.226.67:33037)
0925 15:19:21.481472 (+ 17934us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.481479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.481512 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:21.481575 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.481582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.481770 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.481902 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.484329 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.484336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.484345 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.484360 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.484403 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.484416 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.484594 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.484652 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.484729 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17780,"thread_start_us":17620,"threads_started":1}
I20260925 15:19:21.486505 13880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d3e6bbfafd4cd5bed36378037c203f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:21.486459 14010 tablet_service.cc:1467] Processing CreateTablet for tablet b6d3e6bbfafd4cd5bed36378037c203f (DEFAULT_TABLE table=test-table [id=f7654b0eeefe4db98db7417c426c8dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:21.487680 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d3e6bbfafd4cd5bed36378037c203f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:21.488175 14140 tablet_service.cc:1467] Processing CreateTablet for tablet b6d3e6bbfafd4cd5bed36378037c203f (DEFAULT_TABLE table=test-table [id=f7654b0eeefe4db98db7417c426c8dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:21.489485 14140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d3e6bbfafd4cd5bed36378037c203f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:21.497218 14223 tablet_bootstrap.cc:492] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: Bootstrap starting.
I20260925 15:19:21.500783 14223 tablet_bootstrap.cc:654] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:21.501605 14224 tablet_bootstrap.cc:492] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Bootstrap starting.
I20260925 15:19:21.502070 14223 log.cc:824] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:21.505262 14223 tablet_bootstrap.cc:492] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: No bootstrap required, opened a new log
I20260925 15:19:21.505267 14224 tablet_bootstrap.cc:654] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:21.505393 14226 tablet_bootstrap.cc:492] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Bootstrap starting.
I20260925 15:19:21.505517 14223 ts_tablet_manager.cc:1397] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260925 15:19:21.506484 14224 log.cc:824] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:21.508018 14226 tablet_bootstrap.cc:654] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:21.508876 14226 log.cc:824] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:21.509233 14224 tablet_bootstrap.cc:492] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: No bootstrap required, opened a new log
I20260925 15:19:21.509473 14224 ts_tablet_manager.cc:1397] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:19:21.515008 14226 tablet_bootstrap.cc:492] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: No bootstrap required, opened a new log
I20260925 15:19:21.515247 14226 ts_tablet_manager.cc:1397] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260925 15:19:21.515166 14223 raft_consensus.cc:348] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.515571 14223 raft_consensus.cc:374] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:21.515662 14223 raft_consensus.cc:729] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293c82010c414b9b8afd9190d25f3876, State: Initialized, Role: FOLLOWER
I20260925 15:19:21.516289 14223 consensus_queue.cc:260] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.520535 14224 raft_consensus.cc:348] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.521119 14224 raft_consensus.cc:374] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:21.521232 14224 raft_consensus.cc:729] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ded1eee63a54277ab4d65da49150cf7, State: Initialized, Role: FOLLOWER
I20260925 15:19:21.521981 14224 consensus_queue.cc:260] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.522231 13945 heartbeater.cc:503] Master 127.8.226.126:35501 was elected leader, sending a full tablet report...
I20260925 15:19:21.522934 14075 heartbeater.cc:503] Master 127.8.226.126:35501 was elected leader, sending a full tablet report...
I20260925 15:19:21.523511 14224 ts_tablet_manager.cc:1428] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 15:19:21.524114 14223 ts_tablet_manager.cc:1428] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: Time spent starting tablet: real 0.018s user 0.006s sys 0.012s
I20260925 15:19:21.524431 14226 raft_consensus.cc:348] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.524821 14226 raft_consensus.cc:374] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:21.524924 14226 raft_consensus.cc:729] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db5fd8349944df59d2f1d478baef618, State: Initialized, Role: FOLLOWER
I20260925 15:19:21.525449 14226 consensus_queue.cc:260] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.526600 14205 heartbeater.cc:503] Master 127.8.226.126:35501 was elected leader, sending a full tablet report...
I20260925 15:19:21.527053 14226 ts_tablet_manager.cc:1428] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 15:19:21.529892 14231 raft_consensus.cc:482] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:21.530180 14231 raft_consensus.cc:504] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.532011 14231 leader_election.cc:288] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ded1eee63a54277ab4d65da49150cf7 (127.8.226.66:36813), 293c82010c414b9b8afd9190d25f3876 (127.8.226.65:46407)
I20260925 15:19:21.534371 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.532211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.8.226.67:46953)
0925 15:19:21.532434 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.532453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.532536 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.533054 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.533059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.533081 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.533662 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.533664 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.533681 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.533720 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.534036 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.534041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.534047 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.534147 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.534198 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 15:19:21.534433 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.532358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:46953 (local address 127.8.226.65:46407)
0925 15:19:21.532504 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.532509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.532526 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:21.532692 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.532697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.532762 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.532850 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.533855 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.533861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.533869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.533887 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.533922 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.533939 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.534042 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.534140 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.534195 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:19:21.535269 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.532335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:58457 (local address 127.8.226.66:36813)
0925 15:19:21.532495 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.532502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.533230 (+ 728us) server_negotiation.cc:407] Connection header received
0925 15:19:21.533432 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.533437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.533487 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.533578 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.534712 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.534717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.534721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.534738 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.534773 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.534783 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.534914 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.534985 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.535037 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:19:21.535162 13900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3e6bbfafd4cd5bed36378037c203f" candidate_uuid: "5db5fd8349944df59d2f1d478baef618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293c82010c414b9b8afd9190d25f3876" is_pre_election: true
I20260925 15:19:21.535538 13900 raft_consensus.cc:2454] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5db5fd8349944df59d2f1d478baef618 in term 0.
I20260925 15:19:21.535785 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.532212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36813 (local address 127.8.226.67:58457)
0925 15:19:21.533132 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.533149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:21.533298 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:21.533592 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:21.533596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:21.533621 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:21.534256 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:21.534259 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:21.534275 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:21.534313 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:21.534869 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:21.534872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:21.534878 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:19:21.535000 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 15:19:21.535043 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":30,"thread_start_us":432,"threads_started":1}
I20260925 15:19:21.536301 14095 leader_election.cc:302] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293c82010c414b9b8afd9190d25f3876, 5db5fd8349944df59d2f1d478baef618; no voters:
I20260925 15:19:21.536675 14231 raft_consensus.cc:2781] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:21.536877 14231 raft_consensus.cc:482] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:21.537016 14231 raft_consensus.cc:3037] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:21.536902 14030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3e6bbfafd4cd5bed36378037c203f" candidate_uuid: "5db5fd8349944df59d2f1d478baef618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ded1eee63a54277ab4d65da49150cf7" is_pre_election: true
I20260925 15:19:21.537354 14030 raft_consensus.cc:2454] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5db5fd8349944df59d2f1d478baef618 in term 0.
I20260925 15:19:21.539921 14231 raft_consensus.cc:504] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.540486 14231 leader_election.cc:288] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [CANDIDATE]: Term 1 election: Requested vote from peers 3ded1eee63a54277ab4d65da49150cf7 (127.8.226.66:36813), 293c82010c414b9b8afd9190d25f3876 (127.8.226.65:46407)
I20260925 15:19:21.540817 14030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3e6bbfafd4cd5bed36378037c203f" candidate_uuid: "5db5fd8349944df59d2f1d478baef618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ded1eee63a54277ab4d65da49150cf7"
I20260925 15:19:21.540926 13900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3e6bbfafd4cd5bed36378037c203f" candidate_uuid: "5db5fd8349944df59d2f1d478baef618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293c82010c414b9b8afd9190d25f3876"
I20260925 15:19:21.541038 14030 raft_consensus.cc:3037] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:21.541121 13900 raft_consensus.cc:3037] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:21.544013 14030 raft_consensus.cc:2454] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5db5fd8349944df59d2f1d478baef618 in term 1.
I20260925 15:19:21.544013 13900 raft_consensus.cc:2454] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5db5fd8349944df59d2f1d478baef618 in term 1.
I20260925 15:19:21.544384 14093 leader_election.cc:302] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ded1eee63a54277ab4d65da49150cf7, 5db5fd8349944df59d2f1d478baef618; no voters:
I20260925 15:19:21.544674 14231 raft_consensus.cc:2781] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:21.545173 14231 raft_consensus.cc:686] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 1 LEADER]: Becoming Leader. State: Replica: 5db5fd8349944df59d2f1d478baef618, State: Running, Role: LEADER
I20260925 15:19:21.545655 14231 consensus_queue.cc:237] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } }
I20260925 15:19:21.549744 13757 catalog_manager.cc:5495] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5db5fd8349944df59d2f1d478baef618 (127.8.226.67). New cstate: current_term: 1 leader_uuid: "5db5fd8349944df59d2f1d478baef618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db5fd8349944df59d2f1d478baef618" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33037 } health_report { overall_health: HEALTHY } } }
I20260925 15:19:21.569036 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.567119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.8.226.67:33037)
0925 15:19:21.567309 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.567313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.567329 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:19:21.567493 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.567497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.567564 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.567640 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.568465 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.568469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.568471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.568483 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.568506 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.568516 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.568599 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.568687 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.568746 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:19:21.576336 13900 raft_consensus.cc:1261] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876 [term 1 FOLLOWER]: Refusing update from remote peer 5db5fd8349944df59d2f1d478baef618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:21.576380 14030 raft_consensus.cc:1261] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 1 FOLLOWER]: Refusing update from remote peer 5db5fd8349944df59d2f1d478baef618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:21.577272 14233 consensus_queue.cc:1035] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "293c82010c414b9b8afd9190d25f3876" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:21.577641 14231 consensus_queue.cc:1035] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ded1eee63a54277ab4d65da49150cf7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:21.595062 14237 mvcc.cc:205] Tried to move back new op lower bound from 7333271804208250880 to 7333271804094676992. Current Snapshot: MvccSnapshot[applied={T|T < 7333271804208250880}]
W20260925 15:19:21.602403 13946 tablet.cc:2369] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:21.614547 13758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:21.614812 13758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:21.618477 13758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181c628d4da248ddbd3aac9cb201618d: Sending DeleteTablet for 3 replicas of tablet b6d3e6bbfafd4cd5bed36378037c203f
I20260925 15:19:21.619655 14010 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d3e6bbfafd4cd5bed36378037c203f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:21 UTC) from {username='slave'} at 127.0.0.1:42320
I20260925 15:19:21.619782 13880 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d3e6bbfafd4cd5bed36378037c203f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20260925 15:19:21.620877 14140 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d3e6bbfafd4cd5bed36378037c203f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:21 UTC) from {username='slave'} at 127.0.0.1:46434
I20260925 15:19:21.624997 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13818
I20260925 15:19:21.648587 14249 tablet_replica.cc:330] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: stopping tablet replica
I20260925 15:19:21.649184 14249 raft_consensus.cc:2229] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:21.649541 14247 tablet_replica.cc:330] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: stopping tablet replica
I20260925 15:19:21.649931 14249 raft_consensus.cc:2258] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:21.649992 14247 raft_consensus.cc:2229] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:21.650502 14247 raft_consensus.cc:2258] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:21.652488 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.626372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.8.226.65:46407)
0925 15:19:21.626540 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.626546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.647743 (+ 21197us) server_negotiation.cc:407] Connection header received
0925 15:19:21.647974 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.647980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.648090 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.648191 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.651665 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.651683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.651687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.651707 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.651761 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.651778 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.651888 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.652148 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.652211 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:19:21.653154 14247 ts_tablet_manager.cc:1905] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:21.653450 13940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:21.658250 14249 ts_tablet_manager.cc:1905] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:21.662472 14247 ts_tablet_manager.cc:1918] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:21.662721 14247 log.cc:1196] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/b6d3e6bbfafd4cd5bed36378037c203f
I20260925 15:19:21.663281 14247 ts_tablet_manager.cc:1939] T b6d3e6bbfafd4cd5bed36378037c203f P 3ded1eee63a54277ab4d65da49150cf7: Deleting consensus metadata
I20260925 15:19:21.665071 14249 ts_tablet_manager.cc:1918] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:21.665208 14249 log.cc:1196] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/b6d3e6bbfafd4cd5bed36378037c203f
I20260925 15:19:21.665133 13743 catalog_manager.cc:4841] TS 3ded1eee63a54277ab4d65da49150cf7 (127.8.226.66:36813): tablet b6d3e6bbfafd4cd5bed36378037c203f (table test-table [id=f7654b0eeefe4db98db7417c426c8dcb]) successfully deleted
I20260925 15:19:21.665760 14249 ts_tablet_manager.cc:1939] T b6d3e6bbfafd4cd5bed36378037c203f P 5db5fd8349944df59d2f1d478baef618: Deleting consensus metadata
I20260925 15:19:21.667373 13744 catalog_manager.cc:4841] TS 5db5fd8349944df59d2f1d478baef618 (127.8.226.67:33037): tablet b6d3e6bbfafd4cd5bed36378037c203f (table test-table [id=f7654b0eeefe4db98db7417c426c8dcb]) successfully deleted
I20260925 15:19:21.757611 14248 tablet_replica.cc:330] T b6d3e6bbfafd4cd5bed36378037c203f P 293c82010c414b9b8afd9190d25f3876: stopping tablet replica
I20260925 15:19:21.757824 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13818
I20260925 15:19:21.779297 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13948
W20260925 15:19:21.779307 13745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:46407 (error 108)
I20260925 15:19:21.782284 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.781775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55022)
0925 15:19:21.781929 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.782105 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:19:21.782557 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.780145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.8.226.66:36813)
0925 15:19:21.780331 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.780338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.780480 (+ 142us) server_negotiation.cc:407] Connection header received
0925 15:19:21.780699 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.780704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.780820 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.780925 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.781944 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.781951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.781955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.781973 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.782009 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.782025 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.782141 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.782249 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.782300 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:19:21.782652 13745 catalog_manager.cc:4584] TS 293c82010c414b9b8afd9190d25f3876 (127.8.226.65:46407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6d3e6bbfafd4cd5bed36378037c203f: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
I20260925 15:19:21.783255 14070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:21.836230 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.835705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55030)
0925 15:19:21.835935 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.836045 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:19:21.837368 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.836957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55032)
0925 15:19:21.837122 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.837206 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:19:21.867826 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13948
I20260925 15:19:21.889639 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14078
I20260925 15:19:21.892450 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.890448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.8.226.67:33037)
0925 15:19:21.890670 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:21.890676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:21.890711 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:19:21.890843 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:21.890847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:21.890891 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:21.890989 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:21.891830 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:21.891836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:21.891840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:21.891856 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:21.891894 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:21.891905 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:21.892039 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:21.892117 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:19:21.892174 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 15:19:21.893198 14200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:21.897743 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.897200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55048)
0925 15:19:21.897461 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.897553 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 15:19:21.898665 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.898358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55050)
0925 15:19:21.898482 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.898560 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:19:21.979216 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.978731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55064)
0925 15:19:21.978969 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.979054 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 15:19:21.980281 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.979903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46407 (local address 127.0.0.1:55074)
0925 15:19:21.980093 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.980166 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:19:22.005179 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14078
I20260925 15:19:22.023615 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 13728
I20260925 15:19:22.024771 13789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:22.125478 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 13728
2026-09-25T15:19:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2457 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:19:22.160801 9097 mini_oidc.cc:152] Starting JWKS server
I20260925 15:19:22.163388 9097 webserver.cc:460] Webserver started at https://127.0.0.1:46085/ using document root <none> and password file <none>
I20260925 15:19:22.163715 9097 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:19:22.164325 9097 webserver.cc:460] Webserver started at http://127.0.0.1:44833/ using document root <none> and password file <none>
2026-09-25T15:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:22Z Disabled control of system clock
I20260925 15:19:22.188170 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:45805
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:45805
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46085/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/testchainca.cert with env {}
W20260925 15:19:22.490597 14261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:22.490859 14261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:22.490939 14261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:22.497618 14261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:22.497709 14261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:22.497745 14261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:22.497802 14261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:22.497843 14261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:19:22.497895 14261 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:19:22.497931 14261 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46085/jwks.json
I20260925 15:19:22.506822 14261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:45805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46085/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:22.507969 14261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:22.509181 14261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:22.520375 14267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:22.520665 14269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:22.520853 14266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:22.521874 14261 server_base.cc:1034] running on GCE node
I20260925 15:19:22.522898 14261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:22.524735 14261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:22.525981 14261 hybrid_clock.cc:648] HybridClock initialized: now 1790349562525938 us; error 35 us; skew 500 ppm
I20260925 15:19:22.526360 14261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:22.529040 14261 webserver.cc:460] Webserver started at http://127.8.226.126:39669/ using document root <none> and password file <none>
I20260925 15:19:22.529682 14261 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:22.529811 14261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:22.530102 14261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:22.532859 14261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "a416d28a87674f4ca9b950b225a92bc0"
format_stamp: "Formatted at 2026-09-25 15:19:22 on dist-test-slave-8vw7"
server_key: "73f13e57c2e3e768fd4d02631a8bb05a"
server_key_iv: "f2690a188c07cadf7f7ced6fdc54bea1"
server_key_version: "encryptionkey@0"
I20260925 15:19:22.533713 14261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "a416d28a87674f4ca9b950b225a92bc0"
format_stamp: "Formatted at 2026-09-25 15:19:22 on dist-test-slave-8vw7"
server_key: "73f13e57c2e3e768fd4d02631a8bb05a"
server_key_iv: "f2690a188c07cadf7f7ced6fdc54bea1"
server_key_version: "encryptionkey@0"
I20260925 15:19:22.538242 14261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:19:22.542339 14275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:22.543517 14261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:19:22.543697 14261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "a416d28a87674f4ca9b950b225a92bc0"
format_stamp: "Formatted at 2026-09-25 15:19:22 on dist-test-slave-8vw7"
server_key: "73f13e57c2e3e768fd4d02631a8bb05a"
server_key_iv: "f2690a188c07cadf7f7ced6fdc54bea1"
server_key_version: "encryptionkey@0"
I20260925 15:19:22.543917 14261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:22.571914 14261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:22.572678 14261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:22.572976 14261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:22.589637 14261 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:45805
I20260925 15:19:22.589684 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:45805 every 8 connection(s)
I20260925 15:19:22.590871 14261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:22.594105 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14261
I20260925 15:19:22.594317 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:22.594377 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:22.594946 9097 external_mini_cluster.cc:1428] Setting key 59db147de8c9cd42d767284930a19a70
I20260925 15:19:22.609441 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:22.596918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.8.226.126:45805)
0925 15:19:22.598006 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:22.598019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:22.598066 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:19:22.599747 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:22.599782 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:22.600381 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:22.600821 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:22.601358 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:22.603311 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:22.604659 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:22.605645 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:22.605819 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:22.606863 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:22.606874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:22.606897 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:22.607561 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:22.607648 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:22.607671 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:22.607770 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:22.607912 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:19:22.608236 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":300,"threads_started":1}
I20260925 15:19:22.609467 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0: Bootstrap starting.
I20260925 15:19:22.612540 14327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:22.612854 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:22.613708 14327 log.cc:824] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:22.616185 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0: No bootstrap required, opened a new log
I20260925 15:19:22.622704 14327 raft_consensus.cc:348] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416d28a87674f4ca9b950b225a92bc0" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45805 } }
I20260925 15:19:22.622974 14327 raft_consensus.cc:374] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:22.623031 14327 raft_consensus.cc:729] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a416d28a87674f4ca9b950b225a92bc0, State: Initialized, Role: FOLLOWER
I20260925 15:19:22.623564 14327 consensus_queue.cc:260] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416d28a87674f4ca9b950b225a92bc0" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45805 } }
I20260925 15:19:22.623775 14327 raft_consensus.cc:388] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:22.623891 14327 raft_consensus.cc:482] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:22.624050 14327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:22.625954 14327 raft_consensus.cc:504] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416d28a87674f4ca9b950b225a92bc0" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45805 } }
I20260925 15:19:22.626396 14327 leader_election.cc:302] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a416d28a87674f4ca9b950b225a92bc0; no voters:
I20260925 15:19:22.626875 14327 leader_election.cc:288] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:22.627012 14332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:22.627627 14332 raft_consensus.cc:686] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [term 1 LEADER]: Becoming Leader. State: Replica: a416d28a87674f4ca9b950b225a92bc0, State: Running, Role: LEADER
I20260925 15:19:22.628041 14332 consensus_queue.cc:237] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416d28a87674f4ca9b950b225a92bc0" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45805 } }
I20260925 15:19:22.628669 14327 sys_catalog.cc:564] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:22.630909 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a416d28a87674f4ca9b950b225a92bc0. Latest consensus state: current_term: 1 leader_uuid: "a416d28a87674f4ca9b950b225a92bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416d28a87674f4ca9b950b225a92bc0" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45805 } } }
I20260925 15:19:22.631403 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:22.633983 14333 sys_catalog.cc:455] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a416d28a87674f4ca9b950b225a92bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416d28a87674f4ca9b950b225a92bc0" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45805 } } }
I20260925 15:19:22.634279 14333 sys_catalog.cc:458] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:22.634793 14340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:22.638624 14340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:22.647110 14340 catalog_manager.cc:1347] Generated new cluster ID: 81a2de9ef3044e2e9c7adfafb7db7be1
I20260925 15:19:22.647212 14340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:22.668589 14340 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:22.669951 14340 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:22.676781 14340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a416d28a87674f4ca9b950b225a92bc0: Generated new TSK 0
I20260925 15:19:22.677570 14340 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:19:22.710916 14357 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 15:19:22.711086 14357 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:19:22.712236 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:22.691221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.8.226.126:45805)
0925 15:19:22.691444 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:22.691452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:22.691925 (+ 473us) server_negotiation.cc:407] Connection header received
0925 15:19:22.692179 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:22.692185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:22.692287 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:22.692434 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:19:22.692899 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:22.694003 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:22.695270 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:22.695926 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:22.696062 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:22.696536 (+ 474us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:19:22.711581 (+ 15045us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:19:22.711925 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":106}
W20260925 15:19:22.712350 14330 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:19:22.713531 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:22.691096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:45805 (local address 127.0.0.1:39044)
0925 15:19:22.691828 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:22.691843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:22.691973 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:22.692460 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:22.692464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:22.692473 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:19:22.692777 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:22.692784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:22.694123 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:22.694126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:22.695134 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:22.695144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:22.696107 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:22.696112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:22.696224 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:22.696285 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:19:22.711827 (+ 15542us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:19:22.712069 (+ 242us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.226.126:45805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":400,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.226.126:45805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:19:22.714767 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14261
I20260925 15:19:22.715962 14322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:22.804443 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14261
2026-09-25T15:19:22Z chronyd exiting
I20260925 15:19:22.852815 9097 test_util.cc:175] -----------------------------------------------
I20260925 15:19:22.852988 9097 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349523192742-9097-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:19:22.856340 9097 mini_oidc.cc:152] Starting JWKS server
I20260925 15:19:22.858073 9097 webserver.cc:460] Webserver started at https://127.0.0.1:43079/ using document root <none> and password file <none>
I20260925 15:19:22.858285 9097 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:19:22.858893 9097 webserver.cc:460] Webserver started at http://127.0.0.1:36617/ using document root <none> and password file <none>
2026-09-25T15:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:22Z Disabled control of system clock
I20260925 15:19:22.882866 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:43355
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:43355
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43079/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/testchainca.cert with env {}
W20260925 15:19:23.182510 14364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:23.182762 14364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:23.182855 14364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:23.190477 14364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:23.190572 14364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:23.190608 14364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:23.190662 14364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:23.190711 14364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:19:23.190760 14364 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:19:23.190796 14364 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43079/jwks.json
I20260925 15:19:23.200081 14364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:43355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:43355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43079/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:23.201259 14364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:23.202447 14364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:23.213452 14369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:23.213411 14370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:23.213413 14372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:23.214427 14364 server_base.cc:1034] running on GCE node
I20260925 15:19:23.215134 14364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:23.216372 14364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:23.217568 14364 hybrid_clock.cc:648] HybridClock initialized: now 1790349563217528 us; error 26 us; skew 500 ppm
I20260925 15:19:23.218060 14364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:23.220350 14364 webserver.cc:460] Webserver started at http://127.8.226.126:35159/ using document root <none> and password file <none>
I20260925 15:19:23.221095 14364 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:23.221215 14364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:23.221531 14364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:23.224244 14364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "7b2f87a3fb6e49169c20161265a62f2f"
format_stamp: "Formatted at 2026-09-25 15:19:23 on dist-test-slave-8vw7"
server_key: "32e3737d365789f5feb01c1ea26fd9f3"
server_key_iv: "dfd79726b329f046c472ba2630ff3294"
server_key_version: "encryptionkey@0"
I20260925 15:19:23.225106 14364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "7b2f87a3fb6e49169c20161265a62f2f"
format_stamp: "Formatted at 2026-09-25 15:19:23 on dist-test-slave-8vw7"
server_key: "32e3737d365789f5feb01c1ea26fd9f3"
server_key_iv: "dfd79726b329f046c472ba2630ff3294"
server_key_version: "encryptionkey@0"
I20260925 15:19:23.229930 14364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:23.233104 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:23.234398 14364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:19:23.234594 14364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "7b2f87a3fb6e49169c20161265a62f2f"
format_stamp: "Formatted at 2026-09-25 15:19:23 on dist-test-slave-8vw7"
server_key: "32e3737d365789f5feb01c1ea26fd9f3"
server_key_iv: "dfd79726b329f046c472ba2630ff3294"
server_key_version: "encryptionkey@0"
I20260925 15:19:23.234800 14364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:23.247087 14364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:23.247788 14364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:23.248066 14364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:23.265107 14364 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:43355
I20260925 15:19:23.265084 14429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:43355 every 8 connection(s)
I20260925 15:19:23.266229 14364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:23.266726 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14364
I20260925 15:19:23.266916 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:23.267725 9097 external_mini_cluster.cc:1428] Setting key 18c959571c7da3dfd49a36348845f3d9
I20260925 15:19:23.270450 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:23.282116 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:23.269555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.8.226.126:43355)
0925 15:19:23.270574 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:23.270587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:23.270635 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:19:23.272692 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:23.272715 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:23.273263 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:23.273637 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:23.274171 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.276101 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.277412 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.278355 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.278526 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:23.279549 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:23.279559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:23.279575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:23.280266 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:23.280360 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:23.280389 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:23.280518 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:23.280695 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:19:23.281022 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":317,"threads_started":1}
I20260925 15:19:23.283102 14430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f: Bootstrap starting.
I20260925 15:19:23.285594 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:23.286378 14430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:23.287213 14430 log.cc:824] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:23.290038 14430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f: No bootstrap required, opened a new log
I20260925 15:19:23.297097 14430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f87a3fb6e49169c20161265a62f2f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43355 } }
I20260925 15:19:23.297425 14430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:23.297538 14430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2f87a3fb6e49169c20161265a62f2f, State: Initialized, Role: FOLLOWER
I20260925 15:19:23.298028 14430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f87a3fb6e49169c20161265a62f2f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43355 } }
I20260925 15:19:23.298233 14430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:23.298386 14430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:23.298550 14430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:23.300557 14430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f87a3fb6e49169c20161265a62f2f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43355 } }
I20260925 15:19:23.301126 14430 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2f87a3fb6e49169c20161265a62f2f; no voters:
I20260925 15:19:23.301761 14430 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:23.301905 14435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:23.302645 14435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [term 1 LEADER]: Becoming Leader. State: Replica: 7b2f87a3fb6e49169c20161265a62f2f, State: Running, Role: LEADER
I20260925 15:19:23.303095 14435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f87a3fb6e49169c20161265a62f2f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43355 } }
I20260925 15:19:23.303717 14430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:23.306166 14437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2f87a3fb6e49169c20161265a62f2f. Latest consensus state: current_term: 1 leader_uuid: "7b2f87a3fb6e49169c20161265a62f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f87a3fb6e49169c20161265a62f2f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43355 } } }
I20260925 15:19:23.306138 14436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2f87a3fb6e49169c20161265a62f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f87a3fb6e49169c20161265a62f2f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43355 } } }
I20260925 15:19:23.306591 14437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:23.306622 14436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:23.309607 14442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:23.313849 14442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:23.321870 14442 catalog_manager.cc:1347] Generated new cluster ID: 7630063a24824415913dd9c46effcee6
I20260925 15:19:23.321967 14442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:23.343943 14442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:23.345611 14442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:23.355727 14442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2f87a3fb6e49169c20161265a62f2f: Generated new TSK 0
I20260925 15:19:23.356617 14442 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:19:23.390136 14460 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:19:23.390247 14460 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 15:19:23.391443 14459 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:23.372225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:43355 (local address 127.0.0.1:53436)
0925 15:19:23.372792 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:23.372814 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:23.372918 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:23.373349 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:23.373353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:23.373361 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:19:23.373571 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:23.373578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.374859 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.374862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:23.375922 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:23.375933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.376899 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.376903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:23.377099 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:23.377130 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:19:23.391013 (+ 13883us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:19:23.391178 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.226.126:43355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":327,"threads_started":1}
I20260925 15:19:23.391471 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:23.372313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.8.226.126:43355)
0925 15:19:23.372489 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:23.372495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:23.372893 (+ 398us) server_negotiation.cc:407] Connection header received
0925 15:19:23.373106 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:23.373112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:23.373190 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:23.373337 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:19:23.373714 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.374729 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.376062 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.376751 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.376871 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:23.377392 (+ 521us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:19:23.390824 (+ 13432us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:19:23.391160 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:19:23.391572 14433 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 15:19:23.392453 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14364
I20260925 15:19:23.393569 14425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:23.483386 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14364
2026-09-25T15:19:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (680 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:19:23.536804 9097 mini_oidc.cc:152] Starting JWKS server
I20260925 15:19:23.538677 9097 webserver.cc:460] Webserver started at https://127.0.0.1:37099/ using document root <none> and password file <none>
I20260925 15:19:23.538932 9097 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:19:23.539526 9097 webserver.cc:460] Webserver started at http://127.0.0.1:33053/ using document root <none> and password file <none>
2026-09-25T15:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:23Z Disabled control of system clock
I20260925 15:19:23.566321 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:43777
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:45935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:43777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37099/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:19:23.861075 14467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:23.861338 14467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:23.861413 14467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:23.868201 14467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:23.868296 14467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:23.868333 14467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:23.868381 14467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:23.868424 14467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:19:23.868479 14467 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:19:23.868517 14467 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37099/jwks.json
I20260925 15:19:23.877138 14467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:43777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:43777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37099/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:23.878283 14467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:23.879937 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:23.890883 14472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:23.890931 14475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:23.890954 14473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:23.891883 14467 server_base.cc:1034] running on GCE node
I20260925 15:19:23.892501 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:23.893743 14467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:23.894922 14467 hybrid_clock.cc:648] HybridClock initialized: now 1790349563894888 us; error 24 us; skew 500 ppm
I20260925 15:19:23.895247 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:23.900600 14467 webserver.cc:460] Webserver started at http://127.8.226.126:44155/ using document root <none> and password file <none>
I20260925 15:19:23.901366 14467 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:23.901451 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:23.901798 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:23.904402 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "8095d68d0d034ee4ab676d2938dfa481"
format_stamp: "Formatted at 2026-09-25 15:19:23 on dist-test-slave-8vw7"
server_key: "1c05b4f75bc208d8e2c5cf73e0b01be7"
server_key_iv: "839dbe7ce2f48964a492eabdd502f0da"
server_key_version: "encryptionkey@0"
I20260925 15:19:23.905211 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "8095d68d0d034ee4ab676d2938dfa481"
format_stamp: "Formatted at 2026-09-25 15:19:23 on dist-test-slave-8vw7"
server_key: "1c05b4f75bc208d8e2c5cf73e0b01be7"
server_key_iv: "839dbe7ce2f48964a492eabdd502f0da"
server_key_version: "encryptionkey@0"
I20260925 15:19:23.910097 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:19:23.913205 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:23.914418 14467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:19:23.914574 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "8095d68d0d034ee4ab676d2938dfa481"
format_stamp: "Formatted at 2026-09-25 15:19:23 on dist-test-slave-8vw7"
server_key: "1c05b4f75bc208d8e2c5cf73e0b01be7"
server_key_iv: "839dbe7ce2f48964a492eabdd502f0da"
server_key_version: "encryptionkey@0"
I20260925 15:19:23.914776 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:23.930119 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:23.930864 14467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:23.931167 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:23.948345 14467 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:43777
I20260925 15:19:23.948367 14532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:43777 every 8 connection(s)
I20260925 15:19:23.949671 14467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:23.951537 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14467
I20260925 15:19:23.951740 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:23.952517 9097 external_mini_cluster.cc:1428] Setting key 362f9edd71e822f2c8efe559ca9a31cd
I20260925 15:19:23.953936 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:23.965250 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:23.954350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.8.226.126:43777)
0925 15:19:23.955273 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:23.955284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:23.955320 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:19:23.956768 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:23.956787 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:23.957267 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:23.957577 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:23.958159 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.959467 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.961256 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:23.961864 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:23.961989 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:23.963117 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:23.963124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:23.963138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:23.963643 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:23.963714 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:23.963732 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:23.963813 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:23.964033 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:19:23.964354 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":358,"threads_started":1}
I20260925 15:19:23.966426 14533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481: Bootstrap starting.
I20260925 15:19:23.968746 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:23.969586 14533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:23.970711 14533 log.cc:824] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:23.973466 14533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481: No bootstrap required, opened a new log
I20260925 15:19:23.979956 14533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095d68d0d034ee4ab676d2938dfa481" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43777 } }
I20260925 15:19:23.980374 14533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:23.980479 14533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8095d68d0d034ee4ab676d2938dfa481, State: Initialized, Role: FOLLOWER
I20260925 15:19:23.981040 14533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095d68d0d034ee4ab676d2938dfa481" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43777 } }
I20260925 15:19:23.981254 14533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:23.981395 14533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:23.981557 14533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:23.983362 14533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095d68d0d034ee4ab676d2938dfa481" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43777 } }
I20260925 15:19:23.983807 14533 leader_election.cc:302] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8095d68d0d034ee4ab676d2938dfa481; no voters:
I20260925 15:19:23.984321 14533 leader_election.cc:288] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:23.984442 14538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:23.985359 14538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [term 1 LEADER]: Becoming Leader. State: Replica: 8095d68d0d034ee4ab676d2938dfa481, State: Running, Role: LEADER
I20260925 15:19:23.985862 14538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095d68d0d034ee4ab676d2938dfa481" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43777 } }
I20260925 15:19:23.986155 14533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:23.988365 14540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8095d68d0d034ee4ab676d2938dfa481. Latest consensus state: current_term: 1 leader_uuid: "8095d68d0d034ee4ab676d2938dfa481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095d68d0d034ee4ab676d2938dfa481" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43777 } } }
I20260925 15:19:23.988690 14540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:23.989203 14539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8095d68d0d034ee4ab676d2938dfa481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095d68d0d034ee4ab676d2938dfa481" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 43777 } } }
I20260925 15:19:23.989566 14539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:23.993574 14545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:23.997947 14545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:24.005731 14545 catalog_manager.cc:1347] Generated new cluster ID: fda0e39fb81d4413b220c1377e35b437
I20260925 15:19:24.005822 14545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:24.030606 14545 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:24.031975 14545 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:24.038326 14545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8095d68d0d034ee4ab676d2938dfa481: Generated new TSK 0
I20260925 15:19:24.039377 14545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:24.058533 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:24.050897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.8.226.126:43777)
0925 15:19:24.051088 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:24.051097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:24.051640 (+ 543us) server_negotiation.cc:407] Connection header received
0925 15:19:24.051855 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:24.051859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:24.051940 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:24.052047 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:19:24.052591 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.053345 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.055261 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.055679 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.055767 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.056300 (+ 533us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:19:24.057959 (+ 1659us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37099/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:19:24.058235 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37099/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:19:24.058646 14536 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37099/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 15:19:24.058781 14562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:24.050913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:43777 (local address 127.0.0.1:56268)
0925 15:19:24.051534 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:24.051553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:24.051701 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:24.052095 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:24.052100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:24.052121 (+ 21us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:19:24.052413 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:24.052424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.053482 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.053490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:24.055136 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:24.055153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.055853 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.055862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:24.056043 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.056084 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:19:24.058135 (+ 2051us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37099/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:19:24.058355 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.226.126:43777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37099/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":306,"threads_started":1}
I20260925 15:19:24.060179 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14467
I20260925 15:19:24.061188 14528 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 15:19:24.070194 14563 webserver.cc:533] Webserver: sslize failed:
E20260925 15:19:24.070272 14563 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:19:24.148092 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14467
2026-09-25T15:19:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (665 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:24Z Disabled control of system clock
I20260925 15:19:24.225564 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:42031
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:42031
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/testchainca.cert with env {}
W20260925 15:19:24.520234 14568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:24.520484 14568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:24.520561 14568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:24.526773 14568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:24.526855 14568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:24.526888 14568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:24.526932 14568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:24.526963 14568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:19:24.527017 14568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:19:24.527052 14568 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:19:24.535565 14568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:42031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:24.536625 14568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:24.537794 14568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:24.549961 14574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.550176 14573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.550290 14576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:24.551488 14568 server_base.cc:1034] running on GCE node
I20260925 15:19:24.552157 14568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:24.553450 14568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:24.554627 14568 hybrid_clock.cc:648] HybridClock initialized: now 1790349564554590 us; error 29 us; skew 500 ppm
I20260925 15:19:24.555034 14568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:19:24.557157 14568 webserver.cc:460] Webserver started at http://127.8.226.126:35559/ using document root <none> and password file <none>
I20260925 15:19:24.557765 14568 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:24.557869 14568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:24.558171 14568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:24.560824 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "ccb54e8c00a2445ea2aebdb89fad3c82"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-8vw7"
server_key: "c817757761709ee1fc44169ec071b9fb"
server_key_iv: "f28498f1fca49a853b170136a69ed24e"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.561595 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "ccb54e8c00a2445ea2aebdb89fad3c82"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-8vw7"
server_key: "c817757761709ee1fc44169ec071b9fb"
server_key_iv: "f28498f1fca49a853b170136a69ed24e"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.566062 14568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:19:24.569264 14582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:24.570509 14568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:24.570677 14568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "ccb54e8c00a2445ea2aebdb89fad3c82"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-8vw7"
server_key: "c817757761709ee1fc44169ec071b9fb"
server_key_iv: "f28498f1fca49a853b170136a69ed24e"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.570896 14568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:24.596447 14568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:24.597249 14568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:24.597499 14568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:24.614466 14568 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:42031
I20260925 15:19:24.614488 14633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:42031 every 8 connection(s)
I20260925 15:19:24.615595 14568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:24.618948 14634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:24.620901 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14568
I20260925 15:19:24.621109 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:24.621745 9097 external_mini_cluster.cc:1428] Setting key e23d5f5d4b5ab4cbd66e3cb4ea5b93d1
I20260925 15:19:24.630013 14634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82: Bootstrap starting.
I20260925 15:19:24.633435 14634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:24.634325 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:24.623464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.8.226.126:42031)
0925 15:19:24.624442 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:24.624451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:24.624488 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:19:24.626022 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:24.626043 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:24.626476 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:24.626785 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:24.627379 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.628536 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.630359 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.630962 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.631075 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.632365 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:24.632371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:24.632382 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:24.632842 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:24.632916 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:24.632933 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:24.633068 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:24.633274 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 15:19:24.633530 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":353,"threads_started":1}
I20260925 15:19:24.634362 14634 log.cc:824] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:24.637354 14634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82: No bootstrap required, opened a new log
I20260925 15:19:24.637933 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:24.644382 14634 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42031 } }
I20260925 15:19:24.644716 14634 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:24.644825 14634 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb54e8c00a2445ea2aebdb89fad3c82, State: Initialized, Role: FOLLOWER
I20260925 15:19:24.645414 14634 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42031 } }
I20260925 15:19:24.645638 14634 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:24.645778 14634 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:24.645948 14634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:24.647996 14634 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42031 } }
I20260925 15:19:24.648473 14634 leader_election.cc:302] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccb54e8c00a2445ea2aebdb89fad3c82; no voters:
I20260925 15:19:24.649127 14634 leader_election.cc:288] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:24.649145 14639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:24.649920 14639 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [term 1 LEADER]: Becoming Leader. State: Replica: ccb54e8c00a2445ea2aebdb89fad3c82, State: Running, Role: LEADER
I20260925 15:19:24.650408 14639 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42031 } }
I20260925 15:19:24.651265 14634 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:24.653031 14640 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccb54e8c00a2445ea2aebdb89fad3c82. Latest consensus state: current_term: 1 leader_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42031 } } }
I20260925 15:19:24.653118 14641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb54e8c00a2445ea2aebdb89fad3c82" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 42031 } } }
I20260925 15:19:24.653435 14641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:24.653439 14640 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:24.655071 14646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:24.659602 14646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:24.666952 14646 catalog_manager.cc:1347] Generated new cluster ID: 3e130a2ada8b4b619599c14eeacaba01
I20260925 15:19:24.667094 14646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:24.680356 14646 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:24.682053 14646 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:24.694416 14646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccb54e8c00a2445ea2aebdb89fad3c82: Generated new TSK 0
I20260925 15:19:24.695348 14646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:25.157737 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:24.710406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.8.226.126:42031)
0925 15:19:24.710606 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:24.710613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:24.710928 (+ 315us) server_negotiation.cc:407] Connection header received
0925 15:19:24.711139 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:24.711142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:24.711200 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:24.711315 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:19:24.711835 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.712458 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.714320 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.714698 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.714814 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.715367 (+ 553us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:19:25.157178 (+441811us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:19:25.157504 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51736: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:19:25.157833 14637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51736: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 15:19:25.158098 14663 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:24.710229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:42031 (local address 127.0.0.1:51736)
0925 15:19:24.710830 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:24.710848 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:24.710986 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:24.711361 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:24.711366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:24.711378 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:19:24.711698 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:24.711711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.712591 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.712596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:24.714209 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:24.714222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.714893 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.714900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:24.715077 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.715119 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:19:25.157448 (+442329us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:19:25.157711 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.226.126:42031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":308,"threads_started":1}
I20260925 15:19:25.159406 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14568
I20260925 15:19:25.160530 14629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:25.265812 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14568
2026-09-25T15:19:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1099 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (380 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (385 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:26 dist-test-slave-8vw7 krb5kdc[14678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:26 dist-test-slave-8vw7 krb5kdc[14678](info): set up 2 sockets
Sep 25 15:19:26 dist-test-slave-8vw7 krb5kdc[14678](info): commencing operation
krb5kdc: starting...
W20260925 15:19:27.065495 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:27 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:27.254210 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35765
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:27.556918 14691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:27.557250 14691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:27.557425 14691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:27.565563 14691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:27.565655 14691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:27.565706 14691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:27.565757 14691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:27.565797 14691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:27.574823 14691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:35765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:27.575896 14691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:27.577096 14691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:27.587914 14696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.587954 14697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.587978 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:27.588670 14691 server_base.cc:1034] running on GCE node
I20260925 15:19:27.589437 14691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:27.590692 14691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:27.591890 14691 hybrid_clock.cc:648] HybridClock initialized: now 1790349567591837 us; error 45 us; skew 500 ppm
I20260925 15:19:27.592198 14691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:27 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:27.595033 14691 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:27.596218 14691 webserver.cc:460] Webserver started at http://127.8.226.126:44037/ using document root <none> and password file <none>
I20260925 15:19:27.596766 14691 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:27.596854 14691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:27.597157 14691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:27.599792 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "6d07933db7684d89910dc4f077b4ec8e"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-8vw7"
server_key: "5907c13389bae4682c5f05dcc2560431"
server_key_iv: "e81dda24015cd867036d51c2cd6f25b5"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.600509 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "6d07933db7684d89910dc4f077b4ec8e"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-8vw7"
server_key: "5907c13389bae4682c5f05dcc2560431"
server_key_iv: "e81dda24015cd867036d51c2cd6f25b5"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.605129 14691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:19:27.608091 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.609493 14691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:27.609663 14691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "6d07933db7684d89910dc4f077b4ec8e"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-8vw7"
server_key: "5907c13389bae4682c5f05dcc2560431"
server_key_iv: "e81dda24015cd867036d51c2cd6f25b5"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.609843 14691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:27.627279 14691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:27.630990 14691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:27.631232 14691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:27.648455 14691 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:35765
I20260925 15:19:27.648506 14757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:35765 every 8 connection(s)
I20260925 15:19:27.649864 14691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:27.653457 14758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:27.660039 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14691
I20260925 15:19:27.660264 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:27.661072 9097 external_mini_cluster.cc:1428] Setting key 732deb19a390ce4206752ff6e87c2e1b
I20260925 15:19:27.664351 14758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e: Bootstrap starting.
I20260925 15:19:27.667609 14758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:27.668627 14758 log.cc:824] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:27.671626 14758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e: No bootstrap required, opened a new log
Sep 25 15:19:27 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:27.682202 14758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d07933db7684d89910dc4f077b4ec8e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35765 } }
I20260925 15:19:27.682621 14758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:27.682737 14758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d07933db7684d89910dc4f077b4ec8e, State: Initialized, Role: FOLLOWER
I20260925 15:19:27.683359 14758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d07933db7684d89910dc4f077b4ec8e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35765 } }
I20260925 15:19:27.683599 14758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:27.683768 14758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:27.683936 14758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:27.684860 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.663099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.8.226.126:35765)
0925 15:19:27.663814 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.663825 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.663862 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:19:27.665611 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.665633 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.665979 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.666271 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:27.668010 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.669483 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.671124 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.671807 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.671934 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.676459 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:27.676469 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:27.676487 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:27.676541 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:27.680621 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.681399 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.681420 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.681429 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.681616 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.681951 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.681955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.681959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.682644 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:27.682935 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.683372 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.683795 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":342,"threads_started":1}
I20260925 15:19:27.686360 14758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d07933db7684d89910dc4f077b4ec8e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35765 } }
I20260925 15:19:27.686894 14758 leader_election.cc:302] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d07933db7684d89910dc4f077b4ec8e; no voters:
I20260925 15:19:27.687500 14758 leader_election.cc:288] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:27.687625 14763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:27.688382 14763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [term 1 LEADER]: Becoming Leader. State: Replica: 6d07933db7684d89910dc4f077b4ec8e, State: Running, Role: LEADER
I20260925 15:19:27.688946 14763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d07933db7684d89910dc4f077b4ec8e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35765 } }
I20260925 15:19:27.689654 14758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:27.691732 14764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d07933db7684d89910dc4f077b4ec8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d07933db7684d89910dc4f077b4ec8e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35765 } } }
I20260925 15:19:27.692121 14764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:27.692898 14771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:27.692734 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d07933db7684d89910dc4f077b4ec8e. Latest consensus state: current_term: 1 leader_uuid: "6d07933db7684d89910dc4f077b4ec8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d07933db7684d89910dc4f077b4ec8e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35765 } } }
I20260925 15:19:27.693130 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:27.698220 14771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:27.706447 14771 catalog_manager.cc:1347] Generated new cluster ID: 8fb10af740ec46af8dfbd449c4f5fd81
I20260925 15:19:27.706560 14771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:27.731348 14771 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:27.732532 14771 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:27.743670 14771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d07933db7684d89910dc4f077b4ec8e: Generated new TSK 0
I20260925 15:19:27.744612 14771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:19:27.792119 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35765
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:28.097256 14784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:28.097543 14784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:28.097702 14784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:28.104583 14784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:28.104744 14784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:28.104920 14784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:28.114388 14784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:28.115573 14784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:28.116870 14784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:28.129271 14789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.129254 14790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.129319 14792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.130187 14784 server_base.cc:1034] running on GCE node
I20260925 15:19:28.130767 14784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:28.131662 14784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:28.132884 14784 hybrid_clock.cc:648] HybridClock initialized: now 1790349568132840 us; error 57 us; skew 500 ppm
I20260925 15:19:28.133193 14784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:28 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349568, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:28.136117 14784 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:19:28.137504 14784 webserver.cc:460] Webserver started at http://127.8.226.65:33843/ using document root <none> and password file <none>
I20260925 15:19:28.138132 14784 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:28.138281 14784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:28.138563 14784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:28.141615 14784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "22b4e9ba53f549d39b081fd04a29b602"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-8vw7"
server_key: "8e698e5a1bd6e91cfa22a53a79ea5fd6"
server_key_iv: "b10bf87e97f9df698975690c90999e09"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.142365 14784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "22b4e9ba53f549d39b081fd04a29b602"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-8vw7"
server_key: "8e698e5a1bd6e91cfa22a53a79ea5fd6"
server_key_iv: "b10bf87e97f9df698975690c90999e09"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.147243 14784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:19:28.150413 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.151562 14784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:28.151731 14784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "22b4e9ba53f549d39b081fd04a29b602"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-8vw7"
server_key: "8e698e5a1bd6e91cfa22a53a79ea5fd6"
server_key_iv: "b10bf87e97f9df698975690c90999e09"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.151906 14784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:28.162955 14784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:28.166491 14784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:28.166749 14784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:28.167560 14784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:28.168751 14784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:28.168816 14784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.168890 14784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:28.168931 14784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.200239 14784 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:39073
I20260925 15:19:28.200289 14911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:39073 every 8 connection(s)
I20260925 15:19:28.201494 14784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:28.210147 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14784
I20260925 15:19:28.210382 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:28.210969 9097 external_mini_cluster.cc:1428] Setting key a443a47031fcc336d0088f1053c075fc
Sep 25 15:19:28 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349568, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:28.220616 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.203889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:34725 (local address 127.8.226.126:35765)
0925 15:19:28.204272 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.204276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.205283 (+ 1007us) server_negotiation.cc:407] Connection header received
0925 15:19:28.206957 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.206961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.207023 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.207116 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.208571 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.209364 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.211069 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.211630 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.211723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.215223 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.215227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.215231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:28.215260 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.217769 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:28.218423 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:28.218430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:28.218439 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:28.218548 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:28.218953 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:28.218960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:28.218964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:28.219231 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.219413 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.220056 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.220272 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":194,"threads_started":1}
I20260925 15:19:28.221567 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.204400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35765 (local address 127.8.226.65:34725)
0925 15:19:28.205111 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.205175 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.206729 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.207401 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.207409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.207885 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:28.208415 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.208433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.209514 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.209518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.210927 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.210938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.211858 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.211864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.212093 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.212881 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:28.212902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:28.215003 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:28.217895 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:28.217904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:28.217918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:28.218287 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:28.218661 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:28.218666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:28.218669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:28.218845 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:28.219388 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:28.219396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:28.219717 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.219983 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.220457 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":249,"threads_started":1}
I20260925 15:19:28.223119 14912 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35765
I20260925 15:19:28.223433 14912 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:28.224205 14912 heartbeater.cc:511] Master 127.8.226.126:35765 requested a full tablet report, sending...
I20260925 15:19:28.226521 14723 ts_manager.cc:194] Registered new tserver with Master: 22b4e9ba53f549d39b081fd04a29b602 (127.8.226.65:39073)
I20260925 15:19:28.228681 14723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:34725
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:19:28.255709 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35765
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:28.574467 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:28.574774 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:28.574926 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:28.581712 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:28.581840 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:28.581990 14918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:28.591612 14918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:28.592789 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:28.594053 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:28.606941 14924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.607839 14918 server_base.cc:1034] running on GCE node
W20260925 15:19:28.607999 14926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.608035 14923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.608904 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:28.610088 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:28.611351 14918 hybrid_clock.cc:648] HybridClock initialized: now 1790349568611281 us; error 47 us; skew 500 ppm
I20260925 15:19:28.611780 14918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:28 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349568, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:28.615718 14918 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:19:28.617230 14918 webserver.cc:460] Webserver started at http://127.8.226.66:41111/ using document root <none> and password file <none>
I20260925 15:19:28.617851 14918 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:28.617967 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:28.618250 14918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:28.620939 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "6a26096e0a86414d909dd5ee3d89db6d"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-8vw7"
server_key: "2eeed8b012ff651a8e0fe412b7bbe649"
server_key_iv: "727e040d03f77e0d4b9ab8aa4ebcf2c0"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.621779 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "6a26096e0a86414d909dd5ee3d89db6d"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-8vw7"
server_key: "2eeed8b012ff651a8e0fe412b7bbe649"
server_key_iv: "727e040d03f77e0d4b9ab8aa4ebcf2c0"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.626235 14918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 15:19:28.629240 14933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.630414 14918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:19:28.630565 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "6a26096e0a86414d909dd5ee3d89db6d"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-8vw7"
server_key: "2eeed8b012ff651a8e0fe412b7bbe649"
server_key_iv: "727e040d03f77e0d4b9ab8aa4ebcf2c0"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.630766 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:28.651221 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:28.654974 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:28.655267 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:28.656121 14918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:28.657383 14918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:28.657446 14918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.657573 14918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:28.657617 14918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.689545 14918 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:41231
I20260925 15:19:28.689622 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:41231 every 8 connection(s)
I20260925 15:19:28.690863 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:28.692816 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 14918
I20260925 15:19:28.693035 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:28.693573 9097 external_mini_cluster.cc:1428] Setting key 04c4f29a38d54f30a425ce389d91cc63
Sep 25 15:19:28 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349568, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:28.710510 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.693326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:49105 (local address 127.8.226.126:35765)
0925 15:19:28.693529 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.693535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.694826 (+ 1291us) server_negotiation.cc:407] Connection header received
0925 15:19:28.696796 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.696802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.696882 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.697018 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.698403 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.699099 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.700627 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.701085 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.701213 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.705052 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.705056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.705059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:28.705086 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.707459 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:28.708293 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:28.708297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:28.708299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:28.708413 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:28.708881 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:28.708884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:28.708886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:28.709161 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.709259 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.710135 (+ 876us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.710289 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:19:28.711894 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.693882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35765 (local address 127.8.226.66:49105)
0925 15:19:28.694660 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.694711 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.696537 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.697247 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.697255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.697727 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:28.698249 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.698264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.699247 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.699252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.700473 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.700485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.701310 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.701317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.701574 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.702274 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:28.702295 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:28.704827 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:28.707590 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:28.707600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:28.707615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:28.708185 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:28.708533 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:28.708540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:28.708546 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:28.708780 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:28.709299 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:28.709307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:28.709764 (+ 457us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.710076 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.710620 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:19:28.713228 15046 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35765
I20260925 15:19:28.713507 15046 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:28.714228 15046 heartbeater.cc:511] Master 127.8.226.126:35765 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
I20260925 15:19:28.715559 14723 ts_manager.cc:194] Registered new tserver with Master: 6a26096e0a86414d909dd5ee3d89db6d (127.8.226.66:41231)
I20260925 15:19:28.716296 14723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:49105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:19:28.735952 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35765
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:29.035156 15051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:29.035434 15051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:29.035615 15051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:29.044916 15051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:29.045065 15051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:29.045231 15051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:29.054956 15051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:29.056284 15051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:29.057585 15051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:29.069577 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:29.069519 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:29.069710 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:29.070415 15051 server_base.cc:1034] running on GCE node
I20260925 15:19:29.070973 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:29.071825 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:29.073045 15051 hybrid_clock.cc:648] HybridClock initialized: now 1790349569072975 us; error 40 us; skew 500 ppm
I20260925 15:19:29.073356 15051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.082093 15051 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:19:29.083431 15051 webserver.cc:460] Webserver started at http://127.8.226.67:38779/ using document root <none> and password file <none>
I20260925 15:19:29.084122 15051 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:29.084265 15051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:29.084559 15051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:29.087164 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "4b31da65f6dd41fb9fa50dd83ff417d8"
format_stamp: "Formatted at 2026-09-25 15:19:29 on dist-test-slave-8vw7"
server_key: "25371d9f6d8c99b686c834b007e923e0"
server_key_iv: "b4764e31755ce3a943e5f807a555d1a1"
server_key_version: "encryptionkey@0"
I20260925 15:19:29.087834 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "4b31da65f6dd41fb9fa50dd83ff417d8"
format_stamp: "Formatted at 2026-09-25 15:19:29 on dist-test-slave-8vw7"
server_key: "25371d9f6d8c99b686c834b007e923e0"
server_key_iv: "b4764e31755ce3a943e5f807a555d1a1"
server_key_version: "encryptionkey@0"
I20260925 15:19:29.092586 15051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:19:29.095597 15066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:29.096791 15051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:29.097023 15051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "4b31da65f6dd41fb9fa50dd83ff417d8"
format_stamp: "Formatted at 2026-09-25 15:19:29 on dist-test-slave-8vw7"
server_key: "25371d9f6d8c99b686c834b007e923e0"
server_key_iv: "b4764e31755ce3a943e5f807a555d1a1"
server_key_version: "encryptionkey@0"
I20260925 15:19:29.097223 15051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:29.122201 15051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:29.126425 15051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:29.126701 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:29.127573 15051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:29.128919 15051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:29.129046 15051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:29.129197 15051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:29.129278 15051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:29.161064 15051 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40483
I20260925 15:19:29.161108 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40483 every 8 connection(s)
I20260925 15:19:29.162256 15051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:29.162631 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15051
I20260925 15:19:29.162802 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:29.163435 9097 external_mini_cluster.cc:1428] Setting key 0f1d37b547a6b39cace21e9a2dc309ca
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:29.181052 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.164912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:50683 (local address 127.8.226.126:35765)
0925 15:19:29.165130 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.165136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.166484 (+ 1348us) server_negotiation.cc:407] Connection header received
0925 15:19:29.168219 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.168223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.168281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.168369 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.169775 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.170479 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.171922 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.172296 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.172389 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.175457 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.175462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.175465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.175495 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.178292 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.178915 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.178924 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.178926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.179028 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.179456 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.179467 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.179470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.179715 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.179873 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.180510 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.180723 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 15:19:29.181887 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.165525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35765 (local address 127.8.226.67:50683)
0925 15:19:29.166335 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:29.166381 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:29.167997 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:29.168550 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:29.168557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:29.169088 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:29.169618 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:29.169634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.170607 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.170611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:29.171789 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:29.171819 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.172483 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.172487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:29.172666 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.173278 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:29.173298 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:29.175281 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:29.178419 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:29.178427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:29.178446 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:29.178797 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:29.179128 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:29.179133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:29.179136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:29.179319 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:29.179849 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:29.179856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:29.180147 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:19:29.180395 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 15:19:29.180863 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260925 15:19:29.183096 15179 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35765
I20260925 15:19:29.183347 15179 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:29.184017 15179 heartbeater.cc:511] Master 127.8.226.126:35765 requested a full tablet report, sending...
I20260925 15:19:29.185218 14723 ts_manager.cc:194] Registered new tserver with Master: 4b31da65f6dd41fb9fa50dd83ff417d8 (127.8.226.67:40483)
I20260925 15:19:29.185918 14723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:50683
I20260925 15:19:29.190506 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/test-admin.keytab.
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.215804 9097 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:19:29.220055 15186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:29.218023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.8.226.65:39073)
0925 15:19:29.218563 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.218569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.218594 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:29.218701 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.218707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.218906 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.219018 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.219749 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48742: BlockingRecv error: recv got EOF from 127.0.0.1:48742 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":247,"threads_started":1}
W20260925 15:19:29.220402 9097 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc)
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.222083 9097 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.223387 9097 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:29.232072 14912 heartbeater.cc:503] Master 127.8.226.126:35765 was elected leader, sending a full tablet report...
I20260925 15:19:29.237843 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.224813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.8.226.65:39073)
0925 15:19:29.225005 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.225010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.225165 (+ 155us) server_negotiation.cc:407] Connection header received
0925 15:19:29.225284 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.225288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.225336 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.225406 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.226304 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.227239 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.228280 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.228737 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.228826 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.232510 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.232523 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.232541 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.232573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.235492 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.236062 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.236069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.236071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.236195 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.236514 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.236517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.236520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.236842 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.237097 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.237335 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.237600 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:19:29.243111 15186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:29.241488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.8.226.65:39073)
0925 15:19:29.241678 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.241686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.241924 (+ 238us) server_negotiation.cc:407] Connection header received
0925 15:19:29.242095 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.242106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.242154 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.242244 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.242909 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48756: BlockingRecv error: recv got EOF from 127.0.0.1:48756 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 15:19:29.243335 9097 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.244632 9097 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.245826 9097 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:29.258329 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.246930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.8.226.65:39073)
0925 15:19:29.247112 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.247119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.247366 (+ 247us) server_negotiation.cc:407] Connection header received
0925 15:19:29.247532 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.247537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.247601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.247703 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.248577 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.249305 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.250469 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.250859 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.250980 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.253931 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.253938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.253946 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.253977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.256127 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.256679 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.256685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.256693 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.256799 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.257181 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.257185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.257188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.257424 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.257637 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.257899 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.258124 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:19:29.262082 15186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:29.260465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.8.226.65:39073)
0925 15:19:29.260589 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.260593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.260888 (+ 295us) server_negotiation.cc:407] Connection header received
0925 15:19:29.261073 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.261076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.261132 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.261201 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.261870 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48766: BlockingRecv error: recv got EOF from 127.0.0.1:48766 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 15:19:29.262307 9097 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349523192742-9097-0/krb5kdc/krb5cc)
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.263584 9097 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:29.264699 9097 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:29.276217 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.265798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.8.226.65:39073)
0925 15:19:29.265972 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.265977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.266150 (+ 173us) server_negotiation.cc:407] Connection header received
0925 15:19:29.266241 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.266244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.266286 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.266350 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.267122 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.267800 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.268898 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.269335 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.269418 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.272251 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.272254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.272257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.272281 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.274335 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.274801 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.274804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.274806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.274911 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.275173 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.275175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.275177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.275390 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.275577 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.275766 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.275941 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:19:29.277607 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14784
I20260925 15:19:29.287587 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.278216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.8.226.65:39073)
0925 15:19:29.278348 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.278351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.278557 (+ 206us) server_negotiation.cc:407] Connection header received
0925 15:19:29.278645 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.278648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.278687 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.278776 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.279511 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.280113 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.281232 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.281592 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.281683 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.283385 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.283389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.283392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.283417 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.285746 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.286210 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.286213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.286214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.286325 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.286572 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.286574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.286576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.286792 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.286948 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.287162 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.287311 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:19:29.288157 14907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:29.366983 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14784
I20260925 15:19:29.382450 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14918
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:19:29.397289 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.383280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.8.226.66:41231)
0925 15:19:29.383861 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.383868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.383893 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:29.383958 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.383964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.384188 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.384393 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.385399 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.386360 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.387527 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.388052 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.388183 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.391924 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.391934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.391948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.391984 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.394543 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.395260 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.395267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.395269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.395385 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.395725 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.395730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.395734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.396066 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.396369 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.396711 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.397005 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":278,"threads_started":1}
I20260925 15:19:29.398478 15041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:29.475590 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14918
I20260925 15:19:29.491158 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15051
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:19:29.505805 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:29.492191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.8.226.67:40483)
0925 15:19:29.492895 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:29.492901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:29.492927 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:19:29.493039 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:29.493045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:29.493247 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:29.493386 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:29.494309 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.495260 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.496557 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:29.497269 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:29.497368 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:29.501135 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:29.501145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:29.501157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:29.501184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:29.503575 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.504127 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.504133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.504135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.504234 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:29.504509 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:29.504511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:29.504513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:29.504804 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:29.505071 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:29.505349 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:19:29.505526 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":386,"threads_started":1}
I20260925 15:19:29.507020 15174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:29.587486 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15051
I20260925 15:19:29.602411 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 14691
I20260925 15:19:29.603539 14753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:29.702698 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 14691
2026-09-25T15:19:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3674 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[15206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[15206](info): set up 2 sockets
Sep 25 15:19:29 dist-test-slave-8vw7 krb5kdc[15206](info): commencing operation
krb5kdc: starting...
W20260925 15:19:30.717705 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:30 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab.
Entry for principal oryx/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab.
I20260925 15:19:30.900655 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:33083
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab
--principal=oryx/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:33083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:31.200763 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:31.201088 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:31.201252 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:31.208002 15219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:31.208124 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:31.208209 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:31.208300 15219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:31.208365 15219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:31.217207 15219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:33083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab
--principal=oryx/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:33083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:31.218689 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:31.220050 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:31.231161 15224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.231129 15225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.231102 15227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:31.231987 15219 server_base.cc:1034] running on GCE node
I20260925 15:19:31.232651 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:31.233999 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:31.235208 15219 hybrid_clock.cc:648] HybridClock initialized: now 1790349571235181 us; error 52 us; skew 500 ppm
I20260925 15:19:31.235525 15219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:31 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349571, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:31.238286 15219 init.cc:377] Logged in from keytab as oryx/127.8.226.126@KRBTEST.COM (short username oryx)
I20260925 15:19:31.239482 15219 webserver.cc:460] Webserver started at http://127.8.226.126:45937/ using document root <none> and password file <none>
I20260925 15:19:31.240118 15219 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:31.240262 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:31.240540 15219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:31.243180 15219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "dd5f9acc52c0400fb75f1dd99e5942f4"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "8efe77748c420bfbd43ff1cda5fcee58"
server_key_iv: "72078b3fc4c0687a40d5376b80f19d5c"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.243893 15219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "dd5f9acc52c0400fb75f1dd99e5942f4"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "8efe77748c420bfbd43ff1cda5fcee58"
server_key_iv: "72078b3fc4c0687a40d5376b80f19d5c"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.248633 15219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:19:31.251658 15234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.252863 15219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:31.253062 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "dd5f9acc52c0400fb75f1dd99e5942f4"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "8efe77748c420bfbd43ff1cda5fcee58"
server_key_iv: "72078b3fc4c0687a40d5376b80f19d5c"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.253242 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:31.277485 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:31.281026 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:31.281276 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:31.298036 15219 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:33083
I20260925 15:19:31.298064 15285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:33083 every 8 connection(s)
I20260925 15:19:31.299220 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:31.302644 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:31.306507 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15219
I20260925 15:19:31.306705 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:31.307495 9097 external_mini_cluster.cc:1428] Setting key a4d45d5ea66821d1fe15dbe78fd6c472
I20260925 15:19:31.315565 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Bootstrap starting.
I20260925 15:19:31.319170 15286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:31.320478 15286 log.cc:824] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:19:31 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:31.323257 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: No bootstrap required, opened a new log
I20260925 15:19:31.328883 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.309499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.8.226.126:33083)
0925 15:19:31.310247 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.310258 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.310297 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:19:31.311763 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.311784 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.312157 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.312489 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.314221 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.315378 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.316910 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.317617 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.317753 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.322271 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.322278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.322291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:31.322322 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.325247 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:31.325864 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:31.325870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:31.325875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:31.326026 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:31.326379 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:31.326389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:31.326391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:31.326870 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.327192 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.327554 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.327936 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":367,"threads_started":1}
I20260925 15:19:31.330509 15286 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } }
I20260925 15:19:31.330905 15286 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:31.331012 15286 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5f9acc52c0400fb75f1dd99e5942f4, State: Initialized, Role: FOLLOWER
I20260925 15:19:31.331579 15286 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } }
I20260925 15:19:31.331840 15286 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:31.331956 15286 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:31.332110 15286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:31.334115 15286 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } }
I20260925 15:19:31.334574 15286 leader_election.cc:302] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd5f9acc52c0400fb75f1dd99e5942f4; no voters:
I20260925 15:19:31.335170 15286 leader_election.cc:288] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:31.335294 15291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:31.336265 15291 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 LEADER]: Becoming Leader. State: Replica: dd5f9acc52c0400fb75f1dd99e5942f4, State: Running, Role: LEADER
I20260925 15:19:31.336738 15291 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } }
I20260925 15:19:31.337014 15286 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:31.338881 15293 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5f9acc52c0400fb75f1dd99e5942f4. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } }
I20260925 15:19:31.339138 15293 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:31.339676 15292 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } }
I20260925 15:19:31.339934 15297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:31.339998 15292 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:31.349291 15297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:31.357160 15297 catalog_manager.cc:1347] Generated new cluster ID: 4e34ba29bde74c018c248a281a37c778
I20260925 15:19:31.357277 15297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:31.370590 15297 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:31.371850 15297 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:31.382673 15297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Generated new TSK 0
I20260925 15:19:31.383430 15297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab.
Entry for principal oryx/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab.
I20260925 15:19:31.431772 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33083
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:31.736680 15312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:31.736927 15312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:31.737092 15312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:31.743549 15312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:31.743633 15312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:31.743741 15312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:31.752400 15312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:31.753528 15312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:31.754781 15312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:31.767023 15317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.767176 15320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.767079 15318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:31.767834 15312 server_base.cc:1034] running on GCE node
I20260925 15:19:31.768375 15312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:31.769273 15312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:31.770453 15312 hybrid_clock.cc:648] HybridClock initialized: now 1790349571770399 us; error 54 us; skew 500 ppm
I20260925 15:19:31.770753 15312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:31 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349571, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:31.773667 15312 init.cc:377] Logged in from keytab as oryx/127.8.226.65@KRBTEST.COM (short username oryx)
I20260925 15:19:31.774803 15312 webserver.cc:460] Webserver started at http://127.8.226.65:45919/ using document root <none> and password file <none>
I20260925 15:19:31.775352 15312 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:31.775454 15312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:31.775677 15312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:31.778313 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "acaa02ee2548458fadd257ac9da60c3e"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "ec00effffa8698855ead7c320462a79e"
server_key_iv: "d92b9f7b99c7f4d00e79208700862859"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.778956 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "acaa02ee2548458fadd257ac9da60c3e"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "ec00effffa8698855ead7c320462a79e"
server_key_iv: "d92b9f7b99c7f4d00e79208700862859"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.783671 15312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:19:31.786818 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.788079 15312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:31.788229 15312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "acaa02ee2548458fadd257ac9da60c3e"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "ec00effffa8698855ead7c320462a79e"
server_key_iv: "d92b9f7b99c7f4d00e79208700862859"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.788396 15312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:31.819823 15312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:31.823840 15312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:31.824105 15312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:31.824935 15312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:31.826174 15312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:31.826275 15312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.826423 15312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:31.826510 15312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.857199 15312 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43311
I20260925 15:19:31.857251 15439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43311 every 8 connection(s)
I20260925 15:19:31.858409 15312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:31.859021 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15312
I20260925 15:19:31.859221 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:31.859992 9097 external_mini_cluster.cc:1428] Setting key c62ac5d5d0acb2af748756182e488db4
Sep 25 15:19:31 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349571, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.66@KRBTEST.COM" created.
I20260925 15:19:31.881556 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.860990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:55461 (local address 127.8.226.126:33083)
0925 15:19:31.861455 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.861460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.865731 (+ 4271us) server_negotiation.cc:407] Connection header received
0925 15:19:31.868251 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.868255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.868353 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.868470 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.869974 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.870711 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.872165 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.872524 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.872651 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.876080 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.876085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.876087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:31.876116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.878595 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:31.879298 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:31.879304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:31.879309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:31.879418 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:31.879854 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:31.879859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:31.879862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:31.880121 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.880277 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.880988 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.881209 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
I20260925 15:19:31.882503 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.861631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:55461)
0925 15:19:31.865547 (+ 3916us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.865614 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.867994 (+ 2380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.868696 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.868703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.869245 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:31.869811 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.869827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.870865 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.870877 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.872029 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.872040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.872732 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.872740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.873020 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.873648 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:31.873672 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:31.875883 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:31.878735 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:31.878743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:31.878758 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:31.879172 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:31.879526 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:31.879530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:31.879534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:31.879740 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:31.880257 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:31.880264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:31.880566 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.880877 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.881384 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3593,"thread_start_us":3354,"threads_started":1}
I20260925 15:19:31.886255 15440 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33083
I20260925 15:19:31.886585 15440 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:31.887279 15440 heartbeater.cc:511] Master 127.8.226.126:33083 requested a full tablet report, sending...
I20260925 15:19:31.889348 15251 ts_manager.cc:194] Registered new tserver with Master: acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311)
I20260925 15:19:31.891448 15251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:55461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab.
Entry for principal oryx/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab.
I20260925 15:19:31.904474 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33083
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:32.200222 15446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:32.200568 15446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:32.200662 15446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:32.207338 15446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:32.207441 15446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:32.207543 15446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:32.216137 15446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:32.217288 15446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:32.218433 15446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:32.230726 15454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:32.230844 15451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:32.230808 15452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:32.231829 15446 server_base.cc:1034] running on GCE node
I20260925 15:19:32.232400 15446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:32.233318 15446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:32.234519 15446 hybrid_clock.cc:648] HybridClock initialized: now 1790349572234488 us; error 53 us; skew 500 ppm
I20260925 15:19:32.234792 15446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:32 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349572, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:32.237633 15446 init.cc:377] Logged in from keytab as oryx/127.8.226.66@KRBTEST.COM (short username oryx)
I20260925 15:19:32.238912 15446 webserver.cc:460] Webserver started at http://127.8.226.66:40815/ using document root <none> and password file <none>
I20260925 15:19:32.239535 15446 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:32.239624 15446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:32.239876 15446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:32.242564 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "7a1c9dc0ca6550331f8c1b02f806f33f"
server_key_iv: "8743825ab7a74fe4a4c7f597114703d0"
server_key_version: "encryptionkey@0"
I20260925 15:19:32.243171 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "7a1c9dc0ca6550331f8c1b02f806f33f"
server_key_iv: "8743825ab7a74fe4a4c7f597114703d0"
server_key_version: "encryptionkey@0"
I20260925 15:19:32.247711 15446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:19:32.250790 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:32.251963 15446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 15:19:32.252111 15446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "7a1c9dc0ca6550331f8c1b02f806f33f"
server_key_iv: "8743825ab7a74fe4a4c7f597114703d0"
server_key_version: "encryptionkey@0"
I20260925 15:19:32.252287 15446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:32.265502 15446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:32.268738 15446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:32.269017 15446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:32.269882 15446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:32.271165 15446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:32.271236 15446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:32.271359 15446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:32.271412 15446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:32.305689 15446 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:39809
I20260925 15:19:32.305711 15573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:39809 every 8 connection(s)
I20260925 15:19:32.306949 15446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:32.311419 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15446
I20260925 15:19:32.311586 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:32.312007 9097 external_mini_cluster.cc:1428] Setting key 5036b7eae04f7a1935a63128d22cd915
Sep 25 15:19:32 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349572, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:32.326013 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:32.309305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:39253 (local address 127.8.226.126:33083)
0925 15:19:32.309477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:32.309483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:32.310712 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 15:19:32.313008 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:32.313014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:32.313083 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:32.313182 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:32.314452 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.315168 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.316645 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.317079 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.317201 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:32.320606 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:32.320612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:32.320617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:32.320658 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:32.323240 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:32.323905 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:32.323912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:32.323915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:32.324042 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:32.324439 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:32.324445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:32.324448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:32.324694 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:32.324843 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:32.325519 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 15:19:32.325721 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:19:32.327066 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:32.309781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:39253)
0925 15:19:32.310527 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:32.310580 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:32.312674 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:32.313365 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:32.313372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:32.313817 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:32.314302 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:32.314317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.315333 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.315336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:32.316498 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:32.316509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.317279 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.317286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:32.317517 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:32.318137 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:32.318157 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:32.320407 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:32.323371 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:32.323379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:32.323393 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:32.323791 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:32.324150 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:32.324155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:32.324158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:32.324332 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:32.324821 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:32.324828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:32.325211 (+ 383us) client_negotiation.cc:769] Sending connection context
0925 15:19:32.325447 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 15:19:32.325909 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260925 15:19:32.328588 15574 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33083
I20260925 15:19:32.328881 15574 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:32.329612 15574 heartbeater.cc:511] Master 127.8.226.126:33083 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:19:32.331101 15251 ts_manager.cc:194] Registered new tserver with Master: 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809)
I20260925 15:19:32.331915 15251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:39253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab.
Entry for principal oryx/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab.
I20260925 15:19:32.354274 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33083
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:32.653239 15579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:32.653479 15579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:32.653625 15579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:32.660645 15579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:32.660766 15579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:32.660982 15579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:32.669818 15579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:32.670991 15579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:32.672211 15579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:32.684816 15585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:32.684850 15584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:32.685333 15587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:32.685657 15579 server_base.cc:1034] running on GCE node
I20260925 15:19:32.686210 15579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:32.687088 15579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:32.688297 15579 hybrid_clock.cc:648] HybridClock initialized: now 1790349572688262 us; error 59 us; skew 500 ppm
I20260925 15:19:32.688572 15579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:32 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349572, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:32.691692 15579 init.cc:377] Logged in from keytab as oryx/127.8.226.67@KRBTEST.COM (short username oryx)
I20260925 15:19:32.693130 15579 webserver.cc:460] Webserver started at http://127.8.226.67:33771/ using document root <none> and password file <none>
I20260925 15:19:32.693773 15579 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:32.693874 15579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:32.694089 15579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:32.696585 15579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "1ec949ca51d04e958d33aedc426966b8"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "4a71e74353957b7156afdf76e8fb7b5b"
server_key_iv: "dce1c8d0d7f9a99fb93ef22d06322a28"
server_key_version: "encryptionkey@0"
I20260925 15:19:32.697299 15579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "1ec949ca51d04e958d33aedc426966b8"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "4a71e74353957b7156afdf76e8fb7b5b"
server_key_iv: "dce1c8d0d7f9a99fb93ef22d06322a28"
server_key_version: "encryptionkey@0"
I20260925 15:19:32.701864 15579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:19:32.704833 15594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:32.706116 15579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:19:32.706290 15579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "1ec949ca51d04e958d33aedc426966b8"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "4a71e74353957b7156afdf76e8fb7b5b"
server_key_iv: "dce1c8d0d7f9a99fb93ef22d06322a28"
server_key_version: "encryptionkey@0"
I20260925 15:19:32.706471 15579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:32.733412 15579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:32.737171 15579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:32.737462 15579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:32.738467 15579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:32.739678 15579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:32.739749 15579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:32.739920 15579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:32.739974 15579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:32.772732 15579 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:33529
I20260925 15:19:32.772815 15706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:33529 every 8 connection(s)
I20260925 15:19:32.774016 15579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:32.781649 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15579
I20260925 15:19:32.781827 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:32.782227 9097 external_mini_cluster.cc:1428] Setting key 605bcd6979bf515b7c85f55cc2d15171
Sep 25 15:19:32 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349572, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:32.792289 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:32.776311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:51145 (local address 127.8.226.126:33083)
0925 15:19:32.776475 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:32.776479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:32.777542 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 15:19:32.779168 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:32.779173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:32.779264 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:32.779367 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:32.780773 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.781543 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.783163 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.783703 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.783834 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:32.787174 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:32.787179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:32.787182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:32.787210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:32.789765 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:32.790389 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:32.790392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:32.790395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:32.790501 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:32.790837 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:32.790840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:32.790842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:32.791076 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:32.791203 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:32.791805 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 15:19:32.791956 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:19:32.793310 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:32.776749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:51145)
0925 15:19:32.777398 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:32.777432 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:32.778922 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:32.779577 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:32.779585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:32.780059 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:32.780605 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:32.780621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.781691 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.781700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:32.783030 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:32.783050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:32.784035 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:32.784044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:32.784256 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:32.784895 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:32.784926 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:32.787007 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:32.789876 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:32.789882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:32.789893 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:32.790297 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:32.790587 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:32.790590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:32.790592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:32.790752 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:32.791204 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:32.791209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:32.791469 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 15:19:32.791665 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 15:19:32.792106 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":288,"threads_started":1}
I20260925 15:19:32.794570 15707 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33083
I20260925 15:19:32.794816 15707 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:32.795432 15707 heartbeater.cc:511] Master 127.8.226.126:33083 requested a full tablet report, sending...
I20260925 15:19:32.796834 15251 ts_manager.cc:194] Registered new tserver with Master: 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529)
I20260925 15:19:32.797704 15251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:51145
I20260925 15:19:32.809296 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.226.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.125.keytab.
Entry for principal oryx/127.8.226.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.125.keytab.
I20260925 15:19:32.852715 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:41525
--webserver_interface=127.8.226.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.125.keytab
--principal=oryx/127.8.226.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:33083,127.8.226.125:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:32.894241 15440 heartbeater.cc:503] Master 127.8.226.126:33083 was elected leader, sending a full tablet report...
W20260925 15:19:33.151079 15712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:33.151346 15712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:33.151504 15712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:33.158263 15712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:33.158371 15712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:33.158447 15712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:33.158536 15712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:33.158617 15712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:33.167663 15712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:33083,127.8.226.125:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.125.keytab
--principal=oryx/127.8.226.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.226.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
W20260925 15:19:33.168015 15712 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.226.126:33083,127.8.226.125:41525'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:19:33.168939 15712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:33.170106 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:33.181005 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:33.181111 15717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:33.181079 15718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:33.181823 15712 server_base.cc:1034] running on GCE node
I20260925 15:19:33.182554 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:33.183841 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:33.185102 15712 hybrid_clock.cc:648] HybridClock initialized: now 1790349573185050 us; error 49 us; skew 500 ppm
I20260925 15:19:33.185420 15712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:33.188351 15712 init.cc:377] Logged in from keytab as oryx/127.8.226.125@KRBTEST.COM (short username oryx)
I20260925 15:19:33.189729 15712 webserver.cc:460] Webserver started at http://127.8.226.125:41311/ using document root <none> and password file <none>
I20260925 15:19:33.190379 15712 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:33.190510 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:33.190815 15712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:33.193773 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/instance:
uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "300883c87b2a6810769cf9ea51c61489"
server_key_iv: "70a7eafe3d572fc3d3a03cc0278bd1cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.194500 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal/instance:
uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "300883c87b2a6810769cf9ea51c61489"
server_key_iv: "70a7eafe3d572fc3d3a03cc0278bd1cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.199158 15712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:19:33.202593 15727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:33.203845 15712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:19:33.204106 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "300883c87b2a6810769cf9ea51c61489"
server_key_iv: "70a7eafe3d572fc3d3a03cc0278bd1cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.204299 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:33.233834 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:33.237797 15712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:33.238049 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:33.255532 15712 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.125:41525
I20260925 15:19:33.255550 15778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.125:41525 every 8 connection(s)
I20260925 15:19:33.256901 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
I20260925 15:19:33.259502 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15712
I20260925 15:19:33.260484 15779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:33.267984 15779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.125@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:33.282292 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.261670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.8.226.125:41525)
0925 15:19:33.262428 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.262438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.262474 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:19:33.264502 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.264521 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.264856 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.265270 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.266642 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.267807 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.269020 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.269743 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.269945 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.274575 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.274581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.274591 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.274623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.277605 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.278301 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.278308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.278316 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.278550 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.278889 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.278893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.278896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.279356 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.279649 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.280575 (+ 926us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.281004 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260925 15:19:33.285032 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.270326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.8.226.126:33083)
0925 15:19:33.270504 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.270508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.271272 (+ 764us) server_negotiation.cc:407] Connection header received
0925 15:19:33.271415 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.271418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.271477 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.271551 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.272936 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.273609 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.274927 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.275488 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.275628 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.279046 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.279050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.279053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.279087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.282377 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.283049 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.283053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.283057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.283209 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.283651 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.283654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.283656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.283937 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.284037 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.284502 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.284644 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:19:33.285246 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.270261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:33810)
0925 15:19:33.271098 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.271150 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.271305 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.271642 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.271649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.272459 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.272808 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.272820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.273738 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.273750 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.274774 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.274785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.275681 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.275692 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.275863 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.276500 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.276515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.278884 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.282520 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.282527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.282533 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.282940 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.283330 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.283333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.283335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.283526 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.284049 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.284055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.284236 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.284416 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.284743 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":469,"threads_started":1}
I20260925 15:19:33.286726 15779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.125@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
I20260925 15:19:33.297914 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.287255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48000)
0925 15:19:33.287383 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.287397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.287497 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.287771 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.287789 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.288089 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.288210 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.288217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.289156 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.289161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.290002 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.290012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.290673 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.290678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.290920 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.291331 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.291345 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.293329 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.295817 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.295824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.295828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.296127 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.296450 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.296456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.296459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.296569 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.297022 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.297026 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.297159 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.297288 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.297550 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":19}
I20260925 15:19:33.297991 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.287388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.8.226.125:41525)
0925 15:19:33.287506 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.287511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.287527 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:19:33.287580 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.287584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.287641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.287760 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.288327 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.289026 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.290131 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.290536 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.290665 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.293473 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.293478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.293483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.293517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.295704 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.296243 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.296248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.296252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.296349 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.296662 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.296669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.296672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.296895 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.297047 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.297350 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.297678 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":42}
I20260925 15:19:33.302496 15779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Bootstrap starting.
I20260925 15:19:33.305465 15779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:33.306434 15779 log.cc:824] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:33.309229 15779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: No bootstrap required, opened a new log
I20260925 15:19:33.315800 15779 raft_consensus.cc:348] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } }
I20260925 15:19:33.316146 15779 raft_consensus.cc:374] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:33.316249 15779 raft_consensus.cc:729] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01e1cb1827a4f80b47eed915c06d1ed, State: Initialized, Role: FOLLOWER
I20260925 15:19:33.316727 15779 consensus_queue.cc:260] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } }
I20260925 15:19:33.317535 15785 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } } }
I20260925 15:19:33.317796 15785 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:33.318404 15779 sys_catalog.cc:564] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:19:33.326329 15796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:33.326478 15796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:33.328616 15712 master_runner.cc:428] Detected that this master c01e1cb1827a4f80b47eed915c06d1ed is joining an existing cluster
I20260925 15:19:33.328722 15712 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.226.125:41525
I20260925 15:19:33.334775 15574 heartbeater.cc:503] Master 127.8.226.126:33083 was elected leader, sending a full tablet report...
I20260925 15:19:33.351898 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.332515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.8.226.126:33083)
0925 15:19:33.332710 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.332716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.333310 (+ 594us) server_negotiation.cc:407] Connection header received
0925 15:19:33.333590 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.333598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.333696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.333792 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.337173 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.338634 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.340919 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.341589 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.341686 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.344161 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.344167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.344172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.344206 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.347764 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.348815 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.348822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.348828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.348997 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.349781 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.349788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.349794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.350161 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.350319 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.351208 (+ 889us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.351444 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:19:33.353114 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.332538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:33820)
0925 15:19:33.333206 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.333226 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.333419 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.334033 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.334040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.334884 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.336977 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.336994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.338884 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.338891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.340752 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.340766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.341976 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.341981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.342159 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.342812 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.342836 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.343990 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.347884 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.347891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.347900 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.348686 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.349139 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.349146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.349156 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.349358 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.350395 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.350402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.350616 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.350785 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.352492 (+ 1707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":342,"threads_started":1}
I20260925 15:19:33.354143 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.333376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48004)
0925 15:19:33.339982 (+ 6606us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.340000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.340233 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.340633 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.340638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.341365 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.341654 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.341664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.342981 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.342987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.344549 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.344566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.345770 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.345776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.345925 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.346478 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.346507 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.347483 (+ 976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.351388 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.351392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.351395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.351883 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.352237 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.352261 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.352265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.352401 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.353154 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.353160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.353312 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.353455 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.353714 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6485,"thread_start_us":6689,"threads_started":1}
I20260925 15:19:33.354883 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.333615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.8.226.125:41525)
0925 15:19:33.333788 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.333794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.340064 (+ 6270us) server_negotiation.cc:407] Connection header received
0925 15:19:33.340363 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.340369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.340423 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.340475 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.341790 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.342775 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.344704 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.345448 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.345556 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.347677 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.347683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.347687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.347727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.351267 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.351996 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.352005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.352008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.352131 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.352523 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.352527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.352533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.352912 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.353105 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.354386 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.354568 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for oryx/127.8.226.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.124@KRBTEST.COM" created.
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349571, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.126@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
I20260925 15:19:33.377753 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.360147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48020)
0925 15:19:33.360868 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.360919 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.361098 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.361520 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.361527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.362313 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.362510 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.362518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.363968 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.363972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.365586 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.365604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.366645 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.366654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.366897 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.367526 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.367573 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.370431 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.374689 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.374710 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.374719 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.375251 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.375675 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.375680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.375683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.375846 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.376533 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.376541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.376743 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.376981 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.377325 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":297,"threads_started":1}
I20260925 15:19:33.377821 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.360393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.8.226.125:41525)
0925 15:19:33.361154 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.361160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.361177 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:33.361288 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.361293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.361367 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.361463 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.362663 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.363808 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.365744 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.366468 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.366594 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.370626 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.370634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.370641 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.370680 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.374518 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.375388 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.375393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.375397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.375543 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.375985 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.375992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.375997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.376385 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.376536 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.377138 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.377372 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655}
I20260925 15:19:33.379783 15251 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.226.125:41525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
member_type: NON_VOTER
last_known_addr {
host: "127.8.226.125"
port: 41525
}
attrs {
promote: true
}
}
dest_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4"
cas_config_opid_index: -1
I20260925 15:19:33.380887 15251 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } }
I20260925 15:19:33.384899 15809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:33.385819 15754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:33.387981 15809 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.226.125:41525
I20260925 15:19:33.388696 15712 master.cc:537] Master@127.8.226.125:41525 shutting down...
I20260925 15:19:33.389518 15754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Refusing update from remote peer dd5f9acc52c0400fb75f1dd99e5942f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.124.keytab.
Entry for principal oryx/127.8.226.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.124.keytab.
I20260925 15:19:33.394392 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:43601
--webserver_interface=127.8.226.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.124.keytab
--principal=oryx/127.8.226.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:33.407709 15712 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:33.408217 15712 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:33.408443 15712 tablet_replica.cc:330] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: stopping tablet replica
I20260925 15:19:33.414194 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:33.414527 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:33.420176 15810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 15:19:33.423497 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.423008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48022)
0925 15:19:33.423202 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.423315 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:19:33.423853 15235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.125:41525: connect: Connection refused (error 111)
I20260925 15:19:33.427830 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.427508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48032)
0925 15:19:33.427657 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.427720 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 15:19:33.428040 15235 consensus_peers.cc:487] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 -> Peer c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125:41525): Couldn't send request to peer c01e1cb1827a4f80b47eed915c06d1ed. Status: Network error: Client connection negotiation failed: client connection to 127.8.226.125:41525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:19:33.434056 15712 master.cc:552] Master@127.8.226.125:41525 shutdown complete.
I20260925 15:19:33.434136 15712 master_runner.cc:305] Clearing existing system tablet
I20260925 15:19:33.434473 15712 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:33.434571 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:33.436765 15712 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:19:33.438349 15815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:33.438680 15712 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:33.438939 15712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:33.439014 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "300883c87b2a6810769cf9ea51c61489"
server_key_iv: "70a7eafe3d572fc3d3a03cc0278bd1cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.442272 15712 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:33.446660 15712 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:19:33.446852 15712 log.cc:1196] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:19:33.447217 15712 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Deleting consensus metadata
I20260925 15:19:33.447465 15712 master_runner.cc:315] Copying system tablet from 127.8.226.126:33083
I20260925 15:19:33.449612 15712 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: tablet copy: Beginning tablet copy session from remote peer at address 127.8.226.126:33083
I20260925 15:19:33.462144 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.450260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.8.226.126:33083)
0925 15:19:33.450456 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.450465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.450965 (+ 500us) server_negotiation.cc:407] Connection header received
0925 15:19:33.451167 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.451172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.451261 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.451363 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.452175 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.452868 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.454074 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.454478 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.454592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.456610 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.456616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.456620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.456687 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.459539 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.460220 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.460227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.460230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.460355 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.460723 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.460729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.460733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.461078 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.461262 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.461628 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.461813 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:19:33.462229 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.450260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:33828)
0925 15:19:33.450854 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.450868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.451028 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.451385 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.451390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.451814 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.452025 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.452033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.453052 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.453058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.453945 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.453955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.454613 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.454618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.454790 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.455458 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.455473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.456430 (+ 957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.459687 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.459692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.459696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.460104 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.460463 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.460469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.460472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.460619 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.461239 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.461244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.461399 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.461530 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.461813 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":347,"threads_started":1}
I20260925 15:19:33.463177 15271 tablet_copy_service.cc:140] P dd5f9acc52c0400fb75f1dd99e5942f4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c01e1cb1827a4f80b47eed915c06d1ed ({username='oryx', principal='oryx/127.8.226.125@KRBTEST.COM'} at 127.0.0.1:33828)
I20260925 15:19:33.463393 15271 tablet_copy_service.cc:161] P dd5f9acc52c0400fb75f1dd99e5942f4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c01e1cb1827a4f80b47eed915c06d1ed at {username='oryx', principal='oryx/127.8.226.125@KRBTEST.COM'} at 127.0.0.1:33828: session id = c01e1cb1827a4f80b47eed915c06d1ed-00000000000000000000000000000000
I20260925 15:19:33.466144 15271 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:19:33.467830 15712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:19:33.472018 15712 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: tablet copy: Starting download of 0 data blocks...
I20260925 15:19:33.472265 15712 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: tablet copy: Starting download of 1 WAL segments...
I20260925 15:19:33.474481 15712 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:19:33.477246 15271 tablet_copy_service.cc:342] P dd5f9acc52c0400fb75f1dd99e5942f4: Request end of tablet copy session c01e1cb1827a4f80b47eed915c06d1ed-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.226.125@KRBTEST.COM'} at 127.0.0.1:33828
I20260925 15:19:33.477408 15271 tablet_copy_service.cc:433] P dd5f9acc52c0400fb75f1dd99e5942f4: ending tablet copy session c01e1cb1827a4f80b47eed915c06d1ed-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c01e1cb1827a4f80b47eed915c06d1ed
I20260925 15:19:33.489399 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:33.492252 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:33.492717 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:33.501904 15712 server_base.cc:1034] running on GCE node
W20260925 15:19:33.502249 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:33.502738 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:33.503328 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:33.504542 15712 hybrid_clock.cc:648] HybridClock initialized: now 1790349573504474 us; error 42 us; skew 500 ppm
I20260925 15:19:33.504760 15712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:33.506958 15712 init.cc:377] Logged in from keytab as oryx/127.8.226.125@KRBTEST.COM (short username oryx)
I20260925 15:19:33.508148 15712 webserver.cc:460] Webserver started at http://127.8.226.125:36895/ using document root <none> and password file <none>
I20260925 15:19:33.508464 15712 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:33.508536 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:33.511575 15712 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 15:19:33.513407 15832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:33.513847 15712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:33.513976 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "300883c87b2a6810769cf9ea51c61489"
server_key_iv: "70a7eafe3d572fc3d3a03cc0278bd1cd"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.514145 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:33.549099 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:33.553112 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:33.583163 15712 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.125:41525
I20260925 15:19:33.583209 15883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.125:41525 every 8 connection(s)
I20260925 15:19:33.584065 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
I20260925 15:19:33.589826 15884 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:19:33.594161 15884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Bootstrap starting.
I20260925 15:19:33.610378 15884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:19:33.611104 15884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: Bootstrap complete.
I20260925 15:19:33.612210 15884 raft_consensus.cc:348] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } }
I20260925 15:19:33.612605 15884 raft_consensus.cc:729] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c01e1cb1827a4f80b47eed915c06d1ed, State: Initialized, Role: LEARNER
I20260925 15:19:33.612872 15884 consensus_queue.cc:260] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } }
I20260925 15:19:33.613796 15887 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:33.614100 15887 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: LEARNER
I20260925 15:19:33.614409 15884 sys_catalog.cc:564] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:33.621971 15898 catalog_manager.cc:1259] Loaded cluster ID: 4e34ba29bde74c018c248a281a37c778
I20260925 15:19:33.622056 15898 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: loading cluster ID for follower catalog manager: success
I20260925 15:19:33.625017 15898 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: acquiring CA information for follower catalog manager: success
I20260925 15:19:33.627163 15898 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:19:33.760658 15811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:33.761003 15811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:33.761175 15811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:33.768368 15811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:33.768496 15811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:33.768574 15811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:33.768683 15811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:33.768759 15811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:33.778518 15811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.124.keytab
--principal=oryx/127.8.226.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.226.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:33.779649 15811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:33.780932 15811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:33.791798 15903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:33.791831 15904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:33.792528 15906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:33.793099 15811 server_base.cc:1034] running on GCE node
I20260925 15:19:33.793746 15811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:33.794946 15811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:33.796140 15811 hybrid_clock.cc:648] HybridClock initialized: now 1790349573796109 us; error 45 us; skew 500 ppm
I20260925 15:19:33.796438 15811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:33.799212 15811 init.cc:377] Logged in from keytab as oryx/127.8.226.124@KRBTEST.COM (short username oryx)
I20260925 15:19:33.799998 15707 heartbeater.cc:503] Master 127.8.226.126:33083 was elected leader, sending a full tablet report...
I20260925 15:19:33.800745 15811 webserver.cc:460] Webserver started at http://127.8.226.124:38899/ using document root <none> and password file <none>
I20260925 15:19:33.801381 15811 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:33.801478 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:33.801784 15811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:33.804430 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/instance:
uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "1fa91c73027689ebc1ab741bf43fe62f"
server_key_iv: "248969e27612753be4a5ca4796e18b7f"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.805184 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal/instance:
uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "1fa91c73027689ebc1ab741bf43fe62f"
server_key_iv: "248969e27612753be4a5ca4796e18b7f"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.809801 15811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:19:33.812868 15913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:33.814097 15811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:19:33.814258 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "1fa91c73027689ebc1ab741bf43fe62f"
server_key_iv: "248969e27612753be4a5ca4796e18b7f"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.814445 15811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:33.834422 15811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:33.837971 15811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:33.838270 15811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:33.854969 15811 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.124:43601
I20260925 15:19:33.854971 15964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.124:43601 every 8 connection(s)
I20260925 15:19:33.856343 15811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
I20260925 15:19:33.857321 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 15811
I20260925 15:19:33.861756 15965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:33.871165 15965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.124@KRBTEST.COM
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.124@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:33.883477 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.862764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.8.226.124:43601)
0925 15:19:33.863593 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.863606 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.863652 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:19:33.865195 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.865219 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.865657 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.866063 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.867294 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.869319 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.870552 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.871540 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.871765 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.876230 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.876241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.876255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.876298 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.879482 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.880187 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.880196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.880204 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.880387 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.880716 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.880726 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.880730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.881299 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.881627 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.882119 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.882575 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":406,"threads_started":1}
I20260925 15:19:33.888058 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.873905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:33834)
0925 15:19:33.874714 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.874771 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.874897 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.875239 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.875243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.875834 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.876120 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.876131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.877084 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.877089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.878804 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.878817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.879504 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.879511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.879674 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.880308 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.880352 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.882868 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.886011 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.886020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.886027 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.886403 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.886706 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.886708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.886710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.886822 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.887238 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.887244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.887363 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.887578 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.887809 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"spinlock_wait_cycles":15744,"thread_start_us":471,"threads_started":1}
I20260925 15:19:33.888130 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.874100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.8.226.126:33083)
0925 15:19:33.874314 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.874320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.874858 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:19:33.874998 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.875002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.875084 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.875187 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.876250 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.876919 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.878930 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.879345 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.879428 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.883063 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.883068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.883070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.883099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.885896 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.886495 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.886499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.886501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.886620 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.886901 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.886903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.886905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.887150 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.887249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.887684 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.887834 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:19:33.889336 15965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.124@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
I20260925 15:19:33.901808 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.890118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.8.226.125:41525)
0925 15:19:33.890570 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.890574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.890588 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:19:33.890635 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.890639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.890691 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.890831 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.891542 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.892545 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.893748 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.894196 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.894325 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.896909 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.896914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.896918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.896975 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.899483 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.900101 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.900107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.900111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.900235 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.900581 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.900587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.900590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.900819 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.901009 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.901320 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.901492 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260925 15:19:33.901803 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.889872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48036)
0925 15:19:33.890064 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.890077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.890150 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.890847 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.890851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.891196 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.891375 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.891382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.892656 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.892664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.893604 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.893615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.894354 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.894360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.894539 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.894944 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.894958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.896715 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.899613 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.899618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.899621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.899987 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.900346 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.900352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.900356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.900479 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.900926 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.900931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.901113 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.901237 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.901476 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:19:33.902813 15965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.124@KRBTEST.COM for oryx/127.8.226.124@KRBTEST.COM
I20260925 15:19:33.913600 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.903491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.0.0.1:55328)
0925 15:19:33.903640 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.903656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.903760 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.903965 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.903968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.904245 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.904372 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.904378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.905239 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.905243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.906230 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.906241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.906848 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.906852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.906980 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.907373 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.907387 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.909177 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.911800 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.911803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.911805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.912094 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.912387 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.912390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.912392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.912529 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.912909 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.912912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.913071 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.913187 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.913347 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:19:33.913751 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.903469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.8.226.124:43601)
0925 15:19:33.903598 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.903602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.903728 (+ 126us) server_negotiation.cc:407] Connection header received
0925 15:19:33.903823 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.903826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.903869 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.903932 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.904471 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.905140 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.906342 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.906724 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.906805 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.909318 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.909322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.909325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.909349 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.911703 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.912185 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.912187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.912189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.912299 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.912611 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.912614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.912615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.912822 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.912915 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.913272 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.913453 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":45}
I20260925 15:19:33.918141 15965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Bootstrap starting.
I20260925 15:19:33.921135 15965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:33.922076 15965 log.cc:824] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:33.924355 15965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: No bootstrap required, opened a new log
I20260925 15:19:33.931094 15965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } }
I20260925 15:19:33.931365 15965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:33.931489 15965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9e22b7b2cf44579f9daaca4ccf68a6, State: Initialized, Role: FOLLOWER
I20260925 15:19:33.931981 15965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } }
I20260925 15:19:33.932693 15972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } } }
I20260925 15:19:33.932996 15972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:33.933601 15965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:19:33.942147 15983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:33.942274 15983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:33.945713 15811 master_runner.cc:428] Detected that this master 2a9e22b7b2cf44579f9daaca4ccf68a6 is joining an existing cluster
I20260925 15:19:33.945834 15811 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.226.124:43601
I20260925 15:19:33.968111 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.949051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.8.226.126:33083)
0925 15:19:33.949229 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.949236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.949976 (+ 740us) server_negotiation.cc:407] Connection header received
0925 15:19:33.950265 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.950273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.950368 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.950449 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.952216 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.953609 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.956043 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.956928 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.957067 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.960431 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.960441 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.960448 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.960497 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.964530 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.965734 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.965750 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.965753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.965892 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.966390 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.966394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.966397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.966743 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.966878 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.967543 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.967765 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:19:33.969105 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.949154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:33842)
0925 15:19:33.949864 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.949888 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.950101 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.950689 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.950696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.951595 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.952026 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.952042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.953845 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.953853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.955866 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.955883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.957460 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.957469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.957769 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.958614 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.958645 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.960173 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.964730 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.964737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.964755 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.965569 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.966029 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.966035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.966038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.966242 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.966943 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.966952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.967229 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.968270 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.968553 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":24,"thread_start_us":351,"threads_started":1}
I20260925 15:19:33.972222 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.949111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.8.226.124:43601)
0925 15:19:33.949272 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.949280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.956777 (+ 7497us) server_negotiation.cc:407] Connection header received
0925 15:19:33.956942 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.956946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.957026 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.957106 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.958443 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.959564 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.961512 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.962146 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.962235 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.964454 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.964459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.964464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.964495 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.968064 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.968851 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.968856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.968860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.969028 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.969667 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.969672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.969675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.970690 (+ 1015us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.970821 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.971339 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.971516 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:19:33.973218 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.949491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.0.0.1:55332)
0925 15:19:33.956670 (+ 7179us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.956694 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.956838 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.957242 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.957246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.957999 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.958306 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.958318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.959776 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.959781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.961377 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.961390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.962508 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.962513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.962663 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.963259 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.963278 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.964295 (+ 1017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.968253 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.968261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.968265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.968731 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.969190 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.969195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.969198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.969437 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.970903 (+ 1466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.970909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.971068 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.972420 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.972757 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7086,"thread_start_us":15599,"threads_started":1}
Sep 25 15:19:33 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349571, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.126@KRBTEST.COM for oryx/127.8.226.124@KRBTEST.COM
I20260925 15:19:33.981501 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.950280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.8.226.125:41525)
0925 15:19:33.950438 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.950443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.965763 (+ 15320us) server_negotiation.cc:407] Connection header received
0925 15:19:33.965999 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.966006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.966099 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.966193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.967447 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.968428 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.971397 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.972126 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.972231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.975283 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.975291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.975298 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.975336 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.978734 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.979467 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.979472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.979475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.979609 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.980066 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.980073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.980078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.980349 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.980498 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.980914 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.981151 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:19:33.981578 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.949930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48048)
0925 15:19:33.965660 (+ 15730us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.965676 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.965850 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.966237 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.966244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.966889 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.967281 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.967293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.968858 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.968864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.971217 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.971233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.972729 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.972736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.972942 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.973926 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.973948 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.975031 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.978879 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.978884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.978888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.979332 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.979729 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.979735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.979741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.979929 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.980464 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.980469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.980672 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.980807 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.981116 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15608,"mutex_wait_us":26,"thread_start_us":6386,"threads_started":1}
I20260925 15:19:33.987941 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.974475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.8.226.124:43601)
0925 15:19:33.974742 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.974748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.974761 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:19:33.974840 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.974844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.974891 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.974989 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.975732 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.976574 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.977872 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.978298 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.978427 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.982481 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.982486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.982490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.982519 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.985213 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.986059 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.986068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.986073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.986227 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.986653 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.986657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.986660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.986897 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.986994 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.987449 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.987637 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 15:19:33.988364 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.974144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.0.0.1:55346)
0925 15:19:33.974554 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:33.974569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:33.974661 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:33.975010 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:33.975016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:33.975466 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:33.975597 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.975610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.976743 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.976747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.977707 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:33.977717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.978433 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.978439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:33.978591 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.979073 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:33.979091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:33.982317 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:33.985341 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.985347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.985351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.985919 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.986364 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:33.986369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:33.986373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:33.986541 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:33.987013 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:33.987018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:33.987183 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:19:33.987347 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 15:19:33.987622 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":361,"threads_started":1}
I20260925 15:19:33.988996 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15312
I20260925 15:19:33.989813 15251 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.226.124:43601: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
member_type: NON_VOTER
last_known_addr {
host: "127.8.226.124"
port: 43601
}
attrs {
promote: true
}
}
dest_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4"
cas_config_opid_index: 5
I20260925 15:19:33.990815 15251 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } }
I20260925 15:19:33.995200 15940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:33.999708 15940 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Refusing update from remote peer dd5f9acc52c0400fb75f1dd99e5942f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:19:34.006397 15994 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.65@KRBTEST.COM
I20260925 15:19:34.024572 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.990182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.8.226.65:43311)
0925 15:19:33.990840 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.990850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.990879 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:19:33.990950 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.990959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.991268 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.991448 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.997820 (+ 6372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.999494 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.003406 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.004305 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.011240 (+ 6935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.011543 (+ 303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.011554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.011570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:34.011603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.019453 (+ 7850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.020537 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.020546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.020550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.020693 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.021507 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.021511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.021514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.021923 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.023799 (+ 1876us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.023945 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.024182 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":284,"threads_started":1}
I20260925 15:19:34.025416 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.015735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:48052)
0925 15:19:34.015912 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:34.015929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:34.016023 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:34.017158 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:34.017165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:34.017209 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:34.017476 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.017490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.019344 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.019351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.022007 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.022026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.024426 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.024435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.024709 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.024765 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:19:34.024840 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:19:34.025035 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:19:34.025880 15435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:34.026167 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.016272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.8.226.125:41525)
0925 15:19:34.016482 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.016490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.016511 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:19:34.016617 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.016625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.016737 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.016868 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:34.017697 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.019057 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.022224 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.023892 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.024044 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.024326 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.025663 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.025845 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:19:34.026994 15859 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 LEARNER]: Refusing update from remote peer dd5f9acc52c0400fb75f1dd99e5942f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:19:34.028745 15997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:34.037530 15994 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.226.124:43601
I20260925 15:19:34.038427 15811 master.cc:537] Master@127.8.226.124:43601 shutting down...
I20260925 15:19:34.042455 15997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:19:34.057175 15859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.067408 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.067768 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.068199 15887 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5f9acc52c0400fb75f1dd99e5942f4. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.068454 15887 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: LEARNER
I20260925 15:19:34.068650 15994 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: attempting to promote NON_VOTER c01e1cb1827a4f80b47eed915c06d1ed to VOTER
I20260925 15:19:34.069325 15994 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } }
I20260925 15:19:34.070554 15940 raft_consensus.cc:858] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } }
I20260925 15:19:34.071604 16001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.072136 16001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: LEARNER
I20260925 15:19:34.072230 15858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 LEARNER]: Refusing update from remote peer dd5f9acc52c0400fb75f1dd99e5942f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:19:34.073000 15996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:19:34.075678 15997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.076220 15858 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.081192 16001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.081432 16001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:34.083055 15997 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.083299 15997 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.083568 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.083813 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
W20260925 15:19:34.070937 15940 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } }
W20260925 15:19:34.102092 15919 tablet.cc:2369] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:19:34.112890 15237 consensus_peers.cc:487] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 -> Peer 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124:43601): Couldn't send request to peer 2a9e22b7b2cf44579f9daaca4ccf68a6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:19:34.118527 15940 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:34.122885 16009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:34.123158 16009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 15:19:34.123302 15972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5f9acc52c0400fb75f1dd99e5942f4. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:34.123520 15972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 15:19:34.123544 16006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: NON_VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: true } } }
I20260925 15:19:34.123842 16006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 15:19:34.126466 15811 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 15:19:34.126863 15811 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 15:19:34.127025 15811 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: stopping tablet replica
I20260925 15:19:34.134258 15811 master.cc:552] Master@127.8.226.124:43601 shutdown complete.
I20260925 15:19:34.134428 15811 master_runner.cc:305] Clearing existing system tablet
I20260925 15:19:34.134840 15811 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:34.134994 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:34.137934 15811 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:34.139550 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:34.139853 15811 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:34.140219 15811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:34.140355 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "1fa91c73027689ebc1ab741bf43fe62f"
server_key_iv: "248969e27612753be4a5ca4796e18b7f"
server_key_version: "encryptionkey@0"
I20260925 15:19:34.144455 15811 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:34.151121 15811 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:19:34.151328 15811 log.cc:1196] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:19:34.151859 15811 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Deleting consensus metadata
I20260925 15:19:34.152135 15811 master_runner.cc:315] Copying system tablet from 127.8.226.126:33083
I20260925 15:19:34.154783 15811 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: tablet copy: Beginning tablet copy session from remote peer at address 127.8.226.126:33083
I20260925 15:19:34.171713 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15312
I20260925 15:19:34.177636 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.155981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.8.226.126:33083)
0925 15:19:34.156197 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.156204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.156227 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:19:34.156482 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.156488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.156579 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.156693 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.157743 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.158840 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.160738 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.161369 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.161486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.171189 (+ 9703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.171196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.171202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:34.171243 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.174591 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.175376 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.175384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.175390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.175533 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.175979 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.175986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.175993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.176362 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.176529 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.176935 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.177190 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 15:19:34.177907 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.155562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:33846)
0925 15:19:34.156110 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:34.156134 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:34.156268 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:34.156856 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:34.156861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:34.157354 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:34.157594 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.157605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.158980 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.158985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.160583 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.160599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.161562 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.161569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.161768 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.169773 (+ 8005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:34.169799 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:34.170936 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:34.174789 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.174794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.174798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.175250 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.175682 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.175687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.175691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.175862 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.176492 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:34.176496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:34.176671 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 15:19:34.176817 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 15:19:34.177516 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":677,"threads_started":1}
I20260925 15:19:34.178496 15271 tablet_copy_service.cc:140] P dd5f9acc52c0400fb75f1dd99e5942f4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2a9e22b7b2cf44579f9daaca4ccf68a6 ({username='oryx', principal='oryx/127.8.226.124@KRBTEST.COM'} at 127.0.0.1:33846)
I20260925 15:19:34.178663 15271 tablet_copy_service.cc:161] P dd5f9acc52c0400fb75f1dd99e5942f4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2a9e22b7b2cf44579f9daaca4ccf68a6 at {username='oryx', principal='oryx/127.8.226.124@KRBTEST.COM'} at 127.0.0.1:33846: session id = 2a9e22b7b2cf44579f9daaca4ccf68a6-00000000000000000000000000000000
I20260925 15:19:34.182744 15271 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:19:34.184502 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:19:34.191893 15811 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: tablet copy: Starting download of 0 data blocks...
I20260925 15:19:34.192171 15811 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: tablet copy: Starting download of 1 WAL segments...
I20260925 15:19:34.192358 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:43311
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:33083
--webserver_port=45919
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:34.195094 15811 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:19:34.198177 15271 tablet_copy_service.cc:342] P dd5f9acc52c0400fb75f1dd99e5942f4: Request end of tablet copy session 2a9e22b7b2cf44579f9daaca4ccf68a6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.226.124@KRBTEST.COM'} at 127.0.0.1:33846
I20260925 15:19:34.198309 15271 tablet_copy_service.cc:433] P dd5f9acc52c0400fb75f1dd99e5942f4: ending tablet copy session 2a9e22b7b2cf44579f9daaca4ccf68a6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2a9e22b7b2cf44579f9daaca4ccf68a6
I20260925 15:19:34.209004 15811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:34.211846 16019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:34.212826 16020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:34.219851 16022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:34.226354 15811 server_base.cc:1034] running on GCE node
I20260925 15:19:34.226722 15811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:34.227360 15811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:34.228583 15811 hybrid_clock.cc:648] HybridClock initialized: now 1790349574228528 us; error 32 us; skew 500 ppm
I20260925 15:19:34.228796 15811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:34.230954 15811 init.cc:377] Logged in from keytab as oryx/127.8.226.124@KRBTEST.COM (short username oryx)
I20260925 15:19:34.234127 15811 webserver.cc:460] Webserver started at http://127.8.226.124:42827/ using document root <none> and password file <none>
I20260925 15:19:34.234417 15811 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:34.234506 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:34.237449 15811 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:19:34.239266 16029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:34.239722 15811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:19:34.239874 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-8vw7"
server_key: "1fa91c73027689ebc1ab741bf43fe62f"
server_key_iv: "248969e27612753be4a5ca4796e18b7f"
server_key_version: "encryptionkey@0"
I20260925 15:19:34.240121 15811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:34.277333 15811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:34.281879 15811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:34.331252 15811 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.124:43601
I20260925 15:19:34.331319 16080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.124:43601 every 8 connection(s)
I20260925 15:19:34.332398 15811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
I20260925 15:19:34.338728 16081 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:19:34.342540 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Bootstrap starting.
I20260925 15:19:34.357793 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:19:34.358319 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Bootstrap complete.
I20260925 15:19:34.359493 16081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } }
I20260925 15:19:34.359802 16081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a9e22b7b2cf44579f9daaca4ccf68a6, State: Initialized, Role: LEARNER
I20260925 15:19:34.360049 16081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } }
I20260925 15:19:34.361820 16081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:34.361573 16084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.362075 16084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:19:34.374238 16095 catalog_manager.cc:1259] Loaded cluster ID: 4e34ba29bde74c018c248a281a37c778
I20260925 15:19:34.374334 16095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: loading cluster ID for follower catalog manager: success
I20260925 15:19:34.377894 16095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: acquiring CA information for follower catalog manager: success
I20260925 15:19:34.380263 16095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:19:34.478293 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.472243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.0.0.1:55348)
0925 15:19:34.472503 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:34.472523 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:34.472709 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:34.473637 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:34.473641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:34.473709 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:34.473876 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.473884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.475091 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.475097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.476296 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.476307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.477478 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.477483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.477701 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.477734 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:19:34.477846 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:19:34.477980 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 15:19:34.478289 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.472351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.8.226.124:43601)
0925 15:19:34.473330 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.473335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.473353 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:19:34.473417 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.473420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.473483 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.473613 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:34.474036 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.474959 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.476441 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.477312 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.477445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.477667 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.477869 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.477986 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":740,"threads_started":1}
I20260925 15:19:34.479051 16056 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 15:19:34.479446 16084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5f9acc52c0400fb75f1dd99e5942f4. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.479708 16084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: LEARNER
W20260925 15:19:34.648990 16018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:34.649346 16018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:34.649528 16018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:34.664218 16018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:34.664361 16018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:34.664593 16018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:34.683995 16018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:43311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=45919
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:34.685922 16018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:34.687697 16018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:34.702879 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:34.702857 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:34.703032 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:34.704337 16018 server_base.cc:1034] running on GCE node
I20260925 15:19:34.705201 16018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:34.706437 16018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:34.707702 16018 hybrid_clock.cc:648] HybridClock initialized: now 1790349574707664 us; error 62 us; skew 500 ppm
I20260925 15:19:34.708112 16018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:34.712514 16018 init.cc:377] Logged in from keytab as oryx/127.8.226.65@KRBTEST.COM (short username oryx)
I20260925 15:19:34.714325 16018 webserver.cc:460] Webserver started at http://127.8.226.65:45919/ using document root <none> and password file <none>
I20260925 15:19:34.715266 16018 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:34.715412 16018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:34.722946 16018 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:19:34.726608 16111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:34.728374 16018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:19:34.728564 16018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "acaa02ee2548458fadd257ac9da60c3e"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-8vw7"
server_key: "ec00effffa8698855ead7c320462a79e"
server_key_iv: "d92b9f7b99c7f4d00e79208700862859"
server_key_version: "encryptionkey@0"
I20260925 15:19:34.730230 16018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:34.764401 16018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:34.768019 16018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:34.768234 16018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:34.769063 16018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:34.770325 16018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:34.770406 16018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:34.770478 16018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:34.770550 16018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:34.802373 16018 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43311
I20260925 15:19:34.802436 16223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43311 every 8 connection(s)
I20260925 15:19:34.803933 16018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:34.809489 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 16018
I20260925 15:19:34.809834 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15446
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.124@KRBTEST.COM
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:34.831096 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.809127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:40399 (local address 127.8.226.125:41525)
0925 15:19:34.809635 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.809642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.812308 (+ 2666us) server_negotiation.cc:407] Connection header received
0925 15:19:34.812468 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.812476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.812564 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.812697 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.815814 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.816741 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.818498 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.819046 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.819202 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.823403 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.823410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.823415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:34.823454 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.827134 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.828124 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.828129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.828132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.828284 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.828782 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.828787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.828791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.829273 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.829411 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.830436 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.830667 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":258,"threads_started":1}
I20260925 15:19:34.833182 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.807258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.8.226.65:39481)
0925 15:19:34.808163 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:34.808225 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:34.811525 (+ 3300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:34.813073 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:34.813080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:34.814020 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:34.814928 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.814957 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.816405 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.816412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.818717 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.818729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.820136 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.820142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.820351 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.820940 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:34.820990 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:34.823669 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:34.828217 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.828225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.828231 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.828852 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.829413 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.829421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.829426 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.829996 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.830664 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:34.830671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:34.830833 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:19:34.831004 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 15:19:34.832560 (+ 1556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":928,"spinlock_wait_cycles":90112,"thread_start_us":396,"threads_started":1}
I20260925 15:19:34.833730 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.806664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:39481 (local address 127.8.226.124:43601)
0925 15:19:34.806881 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.806886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.808330 (+ 1444us) server_negotiation.cc:407] Connection header received
0925 15:19:34.811840 (+ 3510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.811850 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.811947 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.812042 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.815140 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.816240 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.818891 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.819722 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.819878 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.823894 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.823899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.823903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:34.823938 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.828025 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.829033 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.829041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.829046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.829260 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.830124 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.830129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.830132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.830479 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.830606 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.831733 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.831972 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:19:34.834504 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.807255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:35755)
0925 15:19:34.808164 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:34.808231 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:34.816658 (+ 8427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:34.817042 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:34.817049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:34.818298 (+ 1249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:34.818608 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.818619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.819952 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.819960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.821640 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.821657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.823688 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.823698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.823904 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.825027 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:34.825054 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:34.828131 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:34.831954 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.831958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.831961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.832411 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.832834 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.832838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.832841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.833024 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.833562 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:34.833566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:34.833709 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 15:19:34.833828 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:19:34.834086 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":1,"spinlock_wait_cycles":46720,"thread_start_us":423,"threads_started":1}
I20260925 15:19:34.834602 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.808739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.8.226.65:40399)
0925 15:19:34.812181 (+ 3442us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:34.812202 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:34.812335 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:34.812977 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:34.812991 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:34.815323 (+ 2332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:34.815660 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.815674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.816896 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.816901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.818342 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:34.818354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.819463 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.819469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:34.819691 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.820487 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:34.820512 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:34.823163 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:34.827325 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.827334 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.827351 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.827971 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.828422 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:34.828429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:34.828434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:34.828656 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:34.829507 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:34.829518 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:34.829957 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 15:19:34.831291 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0925 15:19:34.831706 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3307,"mutex_wait_us":1,"thread_start_us":3175,"threads_started":1}
I20260925 15:19:34.835114 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.806612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:35755 (local address 127.8.226.126:33083)
0925 15:19:34.807189 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.807195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.808320 (+ 1125us) server_negotiation.cc:407] Connection header received
0925 15:19:34.816792 (+ 8472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.816799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.816895 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.817028 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.818755 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.819788 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.822892 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.823512 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.823634 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.828328 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.828336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.828343 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:34.828381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.831798 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.832538 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.832543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.832546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.832695 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.833165 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.833169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.833173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.833423 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.833594 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.834112 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.834340 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":389,"threads_started":1}
I20260925 15:19:34.835503 16226 heartbeater.cc:347] Connected to a master server at 127.8.226.125:41525
I20260925 15:19:34.835891 16226 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:34.836584 16226 heartbeater.cc:511] Master 127.8.226.125:41525 requested a full tablet report, sending...
I20260925 15:19:34.836797 16224 heartbeater.cc:347] Connected to a master server at 127.8.226.124:43601
I20260925 15:19:34.836916 16224 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:34.837357 16224 heartbeater.cc:511] Master 127.8.226.124:43601 requested a full tablet report, sending...
I20260925 15:19:34.837801 16225 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33083
I20260925 15:19:34.837915 16225 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:34.838287 16225 heartbeater.cc:511] Master 127.8.226.126:33083 requested a full tablet report, sending...
I20260925 15:19:34.838964 16046 ts_manager.cc:194] Registered new tserver with Master: acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311)
I20260925 15:19:34.838970 15849 ts_manager.cc:194] Registered new tserver with Master: acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311)
I20260925 15:19:34.839365 15251 ts_manager.cc:194] Re-registered known tserver with Master: acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311)
I20260925 15:19:34.840248 15251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:35755
Sep 25 15:19:34 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.66@KRBTEST.COM
I20260925 15:19:34.849462 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.810801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.8.226.66:39809)
0925 15:19:34.829913 (+ 19112us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.829927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.829958 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:19:34.830029 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.830040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.830328 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.832694 (+ 2366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.832976 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.835083 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.837754 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.840162 (+ 2408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.840260 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.844466 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.844475 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.844485 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:34.844512 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.847280 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.847834 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.847839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.847841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.847949 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:34.848233 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:34.848236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:34.848238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:34.848492 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.848694 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.848926 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.849177 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18893,"thread_start_us":19651,"threads_started":1}
I20260925 15:19:34.850697 15569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:34.931614 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15446
I20260925 15:19:34.947432 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:39809
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:33083
--webserver_port=40815
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:34.974704 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: NON_VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: true } } }
I20260925 15:19:34.975106 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.976516 16238 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: attempting to promote NON_VOTER 2a9e22b7b2cf44579f9daaca4ccf68a6 to VOTER
I20260925 15:19:34.977263 16238 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:34.979676 16056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 LEARNER]: Refusing update from remote peer dd5f9acc52c0400fb75f1dd99e5942f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:19:34.979677 15858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Refusing update from remote peer dd5f9acc52c0400fb75f1dd99e5942f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:19:34.980273 16237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:19:34.980628 15995 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:19:34.982987 16237 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.983599 15858 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.988821 16056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.991314 16237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.991626 16237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.991474 16238 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.991873 16238 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.992018 16237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.992388 16237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.993094 16241 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.993556 16241 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:34.994941 16243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:34.995193 16243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:19:35.267223 16236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:35.267498 16236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:35.267681 16236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:35.274619 16236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:35.274711 16236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:35.274818 16236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:35.283530 16236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:39809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=40815
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:35.284869 16236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:35.286029 16236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:35.298678 16248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:35.298825 16249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:35.301061 16236 server_base.cc:1034] running on GCE node
W20260925 15:19:35.301846 16251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:35.302434 16236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:35.303326 16236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:35.304534 16236 hybrid_clock.cc:648] HybridClock initialized: now 1790349575304489 us; error 40 us; skew 500 ppm
I20260925 15:19:35.304813 16236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:35.307739 16236 init.cc:377] Logged in from keytab as oryx/127.8.226.66@KRBTEST.COM (short username oryx)
I20260925 15:19:35.308990 16236 webserver.cc:460] Webserver started at http://127.8.226.66:40815/ using document root <none> and password file <none>
I20260925 15:19:35.309639 16236 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:35.309798 16236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:35.314775 16236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:19:35.317363 16258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.318634 16236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:35.318790 16236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "7a1c9dc0ca6550331f8c1b02f806f33f"
server_key_iv: "8743825ab7a74fe4a4c7f597114703d0"
server_key_version: "encryptionkey@0"
I20260925 15:19:35.319727 16236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:35.332778 16236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:35.337493 16236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:35.337728 16236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:35.338740 16236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:35.340301 16236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:35.340394 16236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.340480 16236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:35.340545 16236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.375676 16236 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:39809
I20260925 15:19:35.375756 16370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:39809 every 8 connection(s)
I20260925 15:19:35.377099 16236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:35.386576 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 16236
I20260925 15:19:35.386940 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15579
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.124@KRBTEST.COM
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.67@KRBTEST.COM
I20260925 15:19:35.414300 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.380367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:44609 (local address 127.8.226.125:41525)
0925 15:19:35.381313 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.381322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.387679 (+ 6357us) server_negotiation.cc:407] Connection header received
0925 15:19:35.391478 (+ 3799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.391486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.391593 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.391679 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.394837 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.396377 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.399473 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.400075 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.400204 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.406255 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.406262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.406268 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.406314 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.410408 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.411405 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.411412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.411418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.411550 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.412024 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.412029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.412034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.412377 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.412506 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.413526 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.413778 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":650,"threads_started":1}
I20260925 15:19:35.415047 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.386271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:35247 (local address 127.8.226.126:33083)
0925 15:19:35.387318 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.387325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.390782 (+ 3457us) server_negotiation.cc:407] Connection header received
0925 15:19:35.390876 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.390884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.390989 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.391107 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.396079 (+ 4972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.400489 (+ 4410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.403178 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.403864 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.403994 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.408242 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.408247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.408250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.408279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.411863 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.412489 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.412493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.412497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.412634 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.413202 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.413207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.413210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.413556 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.413716 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.414400 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.414651 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":814,"threads_started":1}
I20260925 15:19:35.416236 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.380673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.8.226.66:44609)
0925 15:19:35.387589 (+ 6916us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:35.387606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:35.391306 (+ 3700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:35.392207 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:35.392221 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:35.393520 (+ 1299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:35.394609 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.394639 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.397079 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.397088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.399342 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.399353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.400541 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.400552 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.400975 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.401763 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:35.401794 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:35.405344 (+ 3550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:35.410606 (+ 5262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.410615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.410631 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.411269 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.411669 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.411674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.411680 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.411894 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.412577 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:35.412591 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:35.413062 (+ 471us) client_negotiation.cc:769] Sending connection context
0925 15:19:35.414550 (+ 1488us) client_negotiation.cc:240] Negotiation successful
0925 15:19:35.414948 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6793,"mutex_wait_us":188,"thread_start_us":4504,"threads_started":1}
I20260925 15:19:35.416265 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.386152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:35247)
0925 15:19:35.390427 (+ 4275us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:35.390449 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:35.390604 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:35.392209 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:35.392222 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:35.395474 (+ 3252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:35.395854 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.395869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.400665 (+ 4796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.400670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.402725 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.402739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.404067 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.404074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.404231 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.405522 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:35.405543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:35.408050 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:35.412025 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.412029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.412032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.412373 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.412897 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.412902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.412905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.413078 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.414028 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:35.414032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:35.414135 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 15:19:35.414315 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 15:19:35.414829 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4123,"mutex_wait_us":44,"spinlock_wait_cycles":25344,"thread_start_us":3977,"threads_started":1}
I20260925 15:19:35.417503 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.380731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:51901 (local address 127.8.226.124:43601)
0925 15:19:35.384198 (+ 3467us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.384202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.385722 (+ 1520us) server_negotiation.cc:407] Connection header received
0925 15:19:35.390884 (+ 5162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.390892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.390988 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.391102 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.396733 (+ 5631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.398438 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.401257 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.401895 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.401984 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.407000 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.407008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.407015 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.407060 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.410739 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.412886 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.412891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.412894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.413068 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.413499 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.413503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.413507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.413836 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.413968 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.416620 (+ 2652us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.416827 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3374,"thread_start_us":3258,"threads_started":1}
I20260925 15:19:35.418018 16372 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33083
I20260925 15:19:35.418287 16372 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:35.418431 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.380311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.8.226.66:51901)
0925 15:19:35.385564 (+ 5253us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:35.385608 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:35.390210 (+ 4602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:35.393525 (+ 3315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:35.393533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:35.394028 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:35.394967 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.394981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.399511 (+ 4530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.399516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.401085 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.401098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.402299 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.402305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.402554 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.404193 (+ 1639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:35.404226 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:35.406803 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:35.410906 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.410911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.410914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.412783 (+ 1869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.413186 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.413193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.413207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.413383 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.416157 (+ 2774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:35.416167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:35.416357 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 15:19:35.417758 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0925 15:19:35.418017 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4980,"mutex_wait_us":18,"thread_start_us":8033,"threads_started":1}
I20260925 15:19:35.418959 16373 heartbeater.cc:347] Connected to a master server at 127.8.226.125:41525
I20260925 15:19:35.419088 16373 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:35.419124 16372 heartbeater.cc:511] Master 127.8.226.126:33083 requested a full tablet report, sending...
I20260925 15:19:35.419534 16373 heartbeater.cc:511] Master 127.8.226.125:41525 requested a full tablet report, sending...
I20260925 15:19:35.419955 16371 heartbeater.cc:347] Connected to a master server at 127.8.226.124:43601
I20260925 15:19:35.420101 16371 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:35.420482 16371 heartbeater.cc:511] Master 127.8.226.124:43601 requested a full tablet report, sending...
I20260925 15:19:35.420593 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.387893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.8.226.67:33529)
0925 15:19:35.399220 (+ 11327us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.399228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.399259 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:19:35.399324 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.399331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.399581 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.399744 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.401857 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.403375 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.404881 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.405447 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.405568 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.412078 (+ 6510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.412091 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.412109 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.412151 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.416122 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.417434 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.417442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.417444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.417585 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.418000 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.418004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.418008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.418492 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.418793 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.419893 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.420174 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11170,"thread_start_us":11015,"threads_started":1}
I20260925 15:19:35.420877 15849 ts_manager.cc:194] Registered new tserver with Master: 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809)
I20260925 15:19:35.421417 15251 ts_manager.cc:194] Re-registered known tserver with Master: 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809)
I20260925 15:19:35.421547 16046 ts_manager.cc:194] Registered new tserver with Master: 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809)
I20260925 15:19:35.422262 15251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:35247
I20260925 15:19:35.423933 15702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:35.501972 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15579
I20260925 15:19:35.518534 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:33529
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:33083
--webserver_port=33771
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:35.826817 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:35.827059 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:35.827147 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:35.833985 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:35.834106 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:35.834298 16384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:35.842576 16225 heartbeater.cc:503] Master 127.8.226.126:33083 was elected leader, sending a full tablet report...
I20260925 15:19:35.844847 16384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=33771
--tserver_master_addrs=127.8.226.126:33083,127.8.226.125:41525,127.8.226.124:43601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:35.846084 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:35.847317 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:35.859362 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:35.859474 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:35.859498 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:35.860281 16384 server_base.cc:1034] running on GCE node
I20260925 15:19:35.860848 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:35.861795 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:35.863030 16384 hybrid_clock.cc:648] HybridClock initialized: now 1790349575862967 us; error 42 us; skew 500 ppm
I20260925 15:19:35.863312 16384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:35.866286 16384 init.cc:377] Logged in from keytab as oryx/127.8.226.67@KRBTEST.COM (short username oryx)
I20260925 15:19:35.867553 16384 webserver.cc:460] Webserver started at http://127.8.226.67:33771/ using document root <none> and password file <none>
I20260925 15:19:35.868250 16384 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:35.868402 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:35.873204 16384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:19:35.875563 16399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.876629 16384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:35.876809 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "1ec949ca51d04e958d33aedc426966b8"
format_stamp: "Formatted at 2026-09-25 15:19:32 on dist-test-slave-8vw7"
server_key: "4a71e74353957b7156afdf76e8fb7b5b"
server_key_iv: "dce1c8d0d7f9a99fb93ef22d06322a28"
server_key_version: "encryptionkey@0"
I20260925 15:19:35.877764 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:35.892757 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:35.896262 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:35.896510 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:35.897372 16384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:35.898782 16384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:35.898900 16384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.899300 16384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:35.899401 16384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.931241 16384 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:33529
I20260925 15:19:35.931293 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:33529 every 8 connection(s)
I20260925 15:19:35.932579 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:35.936643 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 16384
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.125@KRBTEST.COM
I20260925 15:19:35.964443 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.936227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:35529 (local address 127.8.226.125:41525)
0925 15:19:35.941751 (+ 5524us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.941758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.941783 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:35.941858 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.941866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.941972 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.942106 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.944292 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.945347 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.948168 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.949060 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.949179 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.955234 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.955244 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.955251 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.955312 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.959748 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.960929 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.960938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.960944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.961164 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.961741 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.961748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.961754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.962165 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.962332 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.963339 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.963603 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5365,"thread_start_us":5248,"threads_started":1}
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.124@KRBTEST.COM
I20260925 15:19:35.968047 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.936677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.8.226.67:35529)
0925 15:19:35.940271 (+ 3594us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:35.940292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:35.940392 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:35.942382 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:35.942395 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:35.943383 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:35.944061 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.944085 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.945550 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.945558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.947966 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.947984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.949583 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.949592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.949985 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.950928 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:35.950967 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:35.954894 (+ 3927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:35.959988 (+ 5094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.960000 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.960021 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.960767 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.961341 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.961347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.961353 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.961595 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.962443 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:35.962453 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:35.962875 (+ 422us) client_negotiation.cc:769] Sending connection context
0925 15:19:35.965115 (+ 2240us) client_negotiation.cc:240] Negotiation successful
0925 15:19:35.966586 (+ 1471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3435,"thread_start_us":3252,"threads_started":1}
I20260925 15:19:35.970108 16514 heartbeater.cc:347] Connected to a master server at 127.8.226.125:41525
I20260925 15:19:35.970499 16514 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:35.971484 16514 heartbeater.cc:511] Master 127.8.226.125:41525 requested a full tablet report, sending...
I20260925 15:19:35.973294 15849 ts_manager.cc:194] Registered new tserver with Master: 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529)
Sep 25 15:19:35 dist-test-slave-8vw7 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:35.983243 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.940452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:39325)
0925 15:19:35.943201 (+ 2749us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:35.943217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:35.950137 (+ 6920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:35.950959 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:35.950966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:35.951687 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:35.952099 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.952114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.956735 (+ 4621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.956743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.962720 (+ 5977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.962737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.969407 (+ 6670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.969415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.969713 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.972292 (+ 2579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:35.972319 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:35.975610 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:35.979437 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.979442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.979445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.979938 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.980861 (+ 923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.980868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.980873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.981081 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.982300 (+ 1219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:35.982306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:35.982457 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:19:35.982614 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 15:19:35.982875 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2634,"thread_start_us":2498,"threads_started":1}
I20260925 15:19:35.983364 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.940565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:39325 (local address 127.8.226.126:33083)
0925 15:19:35.941063 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.941069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.943292 (+ 2223us) server_negotiation.cc:407] Connection header received
0925 15:19:35.950483 (+ 7191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.950492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.950604 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.950691 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.955105 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.956473 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.968194 (+ 11721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.969036 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.969135 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.975812 (+ 6677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.975820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.975827 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.975884 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.979181 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.980045 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.980050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.980053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.980233 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.981804 (+ 1571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.981808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.981811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.982122 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.982258 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.982749 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.982971 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":283,"threads_started":1}
I20260925 15:19:35.984167 16513 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33083
I20260925 15:19:35.984287 16513 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:35.984735 16513 heartbeater.cc:511] Master 127.8.226.126:33083 requested a full tablet report, sending...
I20260925 15:19:35.985553 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.935755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.8.226.67:54379)
0925 15:19:35.936553 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:35.936615 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:35.939399 (+ 2784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:35.943966 (+ 4567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:35.943972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:35.944397 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:35.944633 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.944643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.957970 (+ 13327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.957978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.959821 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:35.959839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.963180 (+ 3341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.963187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:35.963392 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.963990 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:35.964017 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:35.966898 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:35.980869 (+ 13971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.980874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.980878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.981658 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.983553 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:35.983559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:35.983562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:35.983710 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:35.984436 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:35.984442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:35.984593 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:19:35.984730 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:19:35.985081 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"spinlock_wait_cycles":25344,"thread_start_us":2355,"threads_started":1}
I20260925 15:19:35.985816 15251 ts_manager.cc:194] Re-registered known tserver with Master: 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529)
I20260925 15:19:35.986474 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.936337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:54379 (local address 127.8.226.124:43601)
0925 15:19:35.943624 (+ 7287us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.943630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.943647 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:35.943717 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.943722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.943814 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.943920 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.952409 (+ 8489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.953659 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.960074 (+ 6415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.960898 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.963749 (+ 2851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.967149 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.967157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.967163 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.967203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.980739 (+ 13536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.982340 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.982346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.982350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.982487 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.983847 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.983851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.983853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.984303 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.985537 (+ 1234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.985734 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.985967 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":7166,"thread_start_us":7069,"threads_started":1}
I20260925 15:19:35.986936 15251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:39325
I20260925 15:19:35.987368 16512 heartbeater.cc:347] Connected to a master server at 127.8.226.124:43601
I20260925 15:19:35.987471 16512 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:35.987849 16512 heartbeater.cc:511] Master 127.8.226.124:43601 requested a full tablet report, sending...
I20260925 15:19:35.988971 16046 ts_manager.cc:194] Registered new tserver with Master: 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529)
I20260925 15:19:35.993462 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.944323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.8.226.124:43601)
0925 15:19:35.973999 (+ 29676us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.974008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.974029 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:19:35.974085 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.974093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.974178 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.976692 (+ 2514us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.976878 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.978011 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.979792 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.980443 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.981485 (+ 1042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.984300 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.984305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.984310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.984338 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.989896 (+ 5558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.990755 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.990759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.990763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.990905 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.991788 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.991795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.991798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.992126 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.992349 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.992879 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.993096 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29525,"thread_start_us":29981,"threads_started":1}
I20260925 15:19:35.994828 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.945090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.8.226.126:33083)
0925 15:19:35.971561 (+ 26471us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.971567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.972341 (+ 774us) server_negotiation.cc:407] Connection header received
0925 15:19:35.972494 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.972501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.972574 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.972674 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.975031 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.976158 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.978228 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.978823 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.978933 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.982805 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.982813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.982818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.982851 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.991274 (+ 8423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.992205 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.992210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.992213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.992345 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.992775 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.992781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.992787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.993145 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.993393 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.994197 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.994428 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26335,"thread_start_us":26474,"threads_started":1}
I20260925 15:19:35.995096 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.956435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.8.226.125:41525)
0925 15:19:35.965030 (+ 8595us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.965037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.974605 (+ 9568us) server_negotiation.cc:407] Connection header received
0925 15:19:35.975954 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.975958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.976034 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.976089 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.978328 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.979315 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.983689 (+ 4374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:35.984336 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:35.984422 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:35.988262 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.988268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.988272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:35.988306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.991499 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.992247 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.992254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.992257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.992375 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:35.992837 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:35.992842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:35.992845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:35.993200 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.993412 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.994521 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.994747 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8462,"thread_start_us":16189,"threads_started":1}
I20260925 15:19:35.998407 15251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:36.004621 15251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:36.033812 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.022239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.8.226.67:33529)
0925 15:19:36.022953 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.022963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.024228 (+ 1265us) server_negotiation.cc:407] Connection header received
0925 15:19:36.024329 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.024336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.024621 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.024828 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:36.025753 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.027857 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.031073 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.032807 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.032939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.033159 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.033384 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.033522 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":366,"threads_started":1}
I20260925 15:19:36.033854 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.021914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:33529 (local address 127.0.0.1:43150)
0925 15:19:36.023973 (+ 2059us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:36.023990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:36.024108 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:36.025172 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:36.025177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:36.025204 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:36.025547 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.025557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.027984 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.027989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.030916 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.030930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.033033 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.033042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.033190 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.033220 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:19:36.033343 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:19:36.033457 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1948,"mutex_wait_us":6,"thread_start_us":5827,"threads_started":1}
I20260925 15:19:36.035597 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.023423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43311 (local address 127.0.0.1:53196)
0925 15:19:36.028220 (+ 4797us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:36.028243 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:36.028366 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:36.028996 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:36.029001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:36.029023 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:36.029496 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.029514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.031289 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.031296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.033040 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.033053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.034723 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.034728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.034886 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.034916 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:19:36.034973 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:19:36.035081 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4444,"mutex_wait_us":18,"thread_start_us":332,"threads_started":1}
I20260925 15:19:36.036737 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.023746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.8.226.65:43311)
0925 15:19:36.025386 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.025393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.028330 (+ 2937us) server_negotiation.cc:407] Connection header received
0925 15:19:36.028542 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.028550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.028778 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.029433 (+ 655us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:36.029692 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.031144 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.033200 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.034491 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.036030 (+ 1539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.036203 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.036291 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.036440 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":1350,"threads_started":1}
I20260925 15:19:36.036789 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.028111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:39809 (local address 127.0.0.1:35102)
0925 15:19:36.029531 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:36.029547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:36.029666 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:36.030274 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:36.030279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:36.030301 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:36.030670 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.030678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.032265 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.032270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.034236 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.034249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.036136 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.036142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.036305 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.036332 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:19:36.036417 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:19:36.036518 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"thread_start_us":1073,"threads_started":1}
I20260925 15:19:36.037451 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.028370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.8.226.66:39809)
0925 15:19:36.028903 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.028910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.029826 (+ 916us) server_negotiation.cc:407] Connection header received
0925 15:19:36.029916 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.029920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.030110 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.030291 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:36.030825 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.032166 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.034447 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.035719 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.035841 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.036012 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.036507 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.037209 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":301,"threads_started":1}
I20260925 15:19:36.038491 16447 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfca338808d4e97902429749929bb19 (DEFAULT_TABLE table=test-table [id=62277fb9655d4e448530d5be31c9b0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:36.039898 16447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfca338808d4e97902429749929bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:36.040488 16159 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfca338808d4e97902429749929bb19 (DEFAULT_TABLE table=test-table [id=62277fb9655d4e448530d5be31c9b0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:36.042018 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfca338808d4e97902429749929bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:36.043437 16306 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfca338808d4e97902429749929bb19 (DEFAULT_TABLE table=test-table [id=62277fb9655d4e448530d5be31c9b0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:36.044690 16306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfca338808d4e97902429749929bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:36.055953 16548 tablet_bootstrap.cc:492] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Bootstrap starting.
I20260925 15:19:36.057662 16549 tablet_bootstrap.cc:492] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Bootstrap starting.
I20260925 15:19:36.059275 16550 tablet_bootstrap.cc:492] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Bootstrap starting.
I20260925 15:19:36.059738 16548 tablet_bootstrap.cc:654] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:36.061029 16548 log.cc:824] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:36.061296 16549 tablet_bootstrap.cc:654] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:36.062448 16549 log.cc:824] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:36.062518 16550 tablet_bootstrap.cc:654] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:36.063701 16550 log.cc:824] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:36.063987 16548 tablet_bootstrap.cc:492] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: No bootstrap required, opened a new log
I20260925 15:19:36.064229 16548 ts_tablet_manager.cc:1397] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260925 15:19:36.065204 16549 tablet_bootstrap.cc:492] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: No bootstrap required, opened a new log
I20260925 15:19:36.065430 16549 ts_tablet_manager.cc:1397] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:19:36.066880 16550 tablet_bootstrap.cc:492] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: No bootstrap required, opened a new log
I20260925 15:19:36.067108 16550 ts_tablet_manager.cc:1397] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:19:36.073870 16550 raft_consensus.cc:348] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.073935 16548 raft_consensus.cc:348] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.074201 16550 raft_consensus.cc:374] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:36.074314 16550 raft_consensus.cc:729] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0790d9223cfe4e84b208c1c0c9f5e6d1, State: Initialized, Role: FOLLOWER
I20260925 15:19:36.074319 16548 raft_consensus.cc:374] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:36.074397 16548 raft_consensus.cc:729] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec949ca51d04e958d33aedc426966b8, State: Initialized, Role: FOLLOWER
I20260925 15:19:36.074927 16550 consensus_queue.cc:260] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.075054 16548 consensus_queue.cc:260] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.075184 16549 raft_consensus.cc:348] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.075591 16549 raft_consensus.cc:374] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:36.075719 16549 raft_consensus.cc:729] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acaa02ee2548458fadd257ac9da60c3e, State: Initialized, Role: FOLLOWER
I20260925 15:19:36.076341 16549 consensus_queue.cc:260] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.076941 16550 ts_tablet_manager.cc:1428] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260925 15:19:36.077207 16372 heartbeater.cc:503] Master 127.8.226.126:33083 was elected leader, sending a full tablet report...
I20260925 15:19:36.083056 16548 ts_tablet_manager.cc:1428] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20260925 15:19:36.083606 16513 heartbeater.cc:503] Master 127.8.226.126:33083 was elected leader, sending a full tablet report...
I20260925 15:19:36.088150 16549 ts_tablet_manager.cc:1428] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Time spent starting tablet: real 0.023s user 0.012s sys 0.010s
I20260925 15:19:36.093168 16554 raft_consensus.cc:482] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:36.093389 16554 raft_consensus.cc:504] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.094861 16554 leader_election.cc:288] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529), acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311)
I20260925 15:19:36.101397 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.095463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:35515 (local address 127.8.226.65:43311)
0925 15:19:36.095631 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.095635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.095807 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:19:36.096003 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.096008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.096064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.096151 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:36.096646 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.097389 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.099282 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.100282 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.100436 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.100533 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.101026 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.101150 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:19:36.101397 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.095838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:45441 (local address 127.8.226.67:33529)
0925 15:19:36.095979 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.095984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.096000 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:19:36.096055 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.096058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.096103 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.096211 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:36.096722 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.097428 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.099268 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.100371 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.100538 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.100602 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.101026 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.101150 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:19:36.102128 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.095015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:33529 (local address 127.8.226.66:45441)
0925 15:19:36.095690 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:36.095726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:36.095851 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:36.096226 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:36.096230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:36.096269 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:36.096577 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.096585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.097553 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.097558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.099127 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.099139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.100573 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.100580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.100819 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.100854 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:19:36.101543 (+ 689us) client_negotiation.cc:240] Negotiation successful
0925 15:19:36.101697 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":152,"spinlock_wait_cycles":14976,"thread_start_us":347,"threads_started":1}
I20260925 15:19:36.102147 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.095015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43311 (local address 127.8.226.66:35515)
0925 15:19:36.095690 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:36.095726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:36.095851 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:36.096167 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:36.096171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:36.096195 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:36.096510 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.096518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.097499 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.097505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.099133 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:36.099144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.100654 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.100661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:36.100833 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.100855 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:19:36.101543 (+ 688us) client_negotiation.cc:240] Negotiation successful
0925 15:19:36.101697 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":73,"spinlock_wait_cycles":4480,"thread_start_us":853,"threads_started":1}
I20260925 15:19:36.103003 16179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfca338808d4e97902429749929bb19" candidate_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acaa02ee2548458fadd257ac9da60c3e" is_pre_election: true
I20260925 15:19:36.103082 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfca338808d4e97902429749929bb19" candidate_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec949ca51d04e958d33aedc426966b8" is_pre_election: true
I20260925 15:19:36.103427 16179 raft_consensus.cc:2454] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0790d9223cfe4e84b208c1c0c9f5e6d1 in term 0.
I20260925 15:19:36.103528 16467 raft_consensus.cc:2454] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0790d9223cfe4e84b208c1c0c9f5e6d1 in term 0.
I20260925 15:19:36.104115 16261 leader_election.cc:302] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0790d9223cfe4e84b208c1c0c9f5e6d1, acaa02ee2548458fadd257ac9da60c3e; no voters:
I20260925 15:19:36.104498 16554 raft_consensus.cc:2781] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:36.104658 16554 raft_consensus.cc:482] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:36.104771 16554 raft_consensus.cc:3037] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:36.107033 16554 raft_consensus.cc:504] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.107690 16554 leader_election.cc:288] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529), acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311)
I20260925 15:19:36.107960 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfca338808d4e97902429749929bb19" candidate_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec949ca51d04e958d33aedc426966b8"
I20260925 15:19:36.108136 16467 raft_consensus.cc:3037] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:36.108107 16179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfca338808d4e97902429749929bb19" candidate_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acaa02ee2548458fadd257ac9da60c3e"
I20260925 15:19:36.108297 16179 raft_consensus.cc:3037] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:36.110342 16467 raft_consensus.cc:2454] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0790d9223cfe4e84b208c1c0c9f5e6d1 in term 1.
I20260925 15:19:36.110492 16179 raft_consensus.cc:2454] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0790d9223cfe4e84b208c1c0c9f5e6d1 in term 1.
I20260925 15:19:36.110715 16259 leader_election.cc:302] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0790d9223cfe4e84b208c1c0c9f5e6d1, 1ec949ca51d04e958d33aedc426966b8; no voters:
I20260925 15:19:36.110989 16554 raft_consensus.cc:2781] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:36.111424 16554 raft_consensus.cc:686] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 0790d9223cfe4e84b208c1c0c9f5e6d1, State: Running, Role: LEADER
I20260925 15:19:36.111869 16554 consensus_queue.cc:237] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } }
I20260925 15:19:36.116621 15251 catalog_manager.cc:5495] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } health_report { overall_health: UNKNOWN } } }
W20260925 15:19:36.149441 16374 tablet.cc:2369] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:36.154654 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.150184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.8.226.66:39809)
0925 15:19:36.150375 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.150379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.150396 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:36.150549 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.150552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.150622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.150755 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:36.151052 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.151725 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.152886 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.153315 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.153396 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.153768 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:36.154210 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:36.154339 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.154409 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.154467 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:19:36.162174 16467 raft_consensus.cc:1261] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Refusing update from remote peer 0790d9223cfe4e84b208c1c0c9f5e6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:36.162230 16179 raft_consensus.cc:1261] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Refusing update from remote peer 0790d9223cfe4e84b208c1c0c9f5e6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:36.163136 16559 consensus_queue.cc:1035] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:36.163453 16554 consensus_queue.cc:1035] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:36.176822 16563 mvcc.cc:205] Tried to move back new op lower bound from 7333271863951015936 to 7333271863758086144. Current Snapshot: MvccSnapshot[applied={T|T < 7333271863951015936}]
I20260925 15:19:36.188758 16562 mvcc.cc:205] Tried to move back new op lower bound from 7333271863951015936 to 7333271863758086144. Current Snapshot: MvccSnapshot[applied={T|T < 7333271863951015936 or (T in {7333271863951015936})}]
I20260925 15:19:36.196174 15250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:36.196440 15250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:36.204071 15250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd5f9acc52c0400fb75f1dd99e5942f4: Sending DeleteTablet for 3 replicas of tablet 6cfca338808d4e97902429749929bb19
I20260925 15:19:36.205245 16159 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfca338808d4e97902429749929bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:36 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:53196
I20260925 15:19:36.205245 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfca338808d4e97902429749929bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:36 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:43150
I20260925 15:19:36.206043 16572 tablet_replica.cc:330] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: stopping tablet replica
I20260925 15:19:36.206058 16306 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfca338808d4e97902429749929bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:36 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:35102
I20260925 15:19:36.206476 16572 raft_consensus.cc:2229] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:36.207020 16572 raft_consensus.cc:2258] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:36.211094 16572 ts_tablet_manager.cc:1905] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:36.217665 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15219
I20260925 15:19:36.218909 15281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:36.220276 16572 ts_tablet_manager.cc:1918] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:36.220474 16572 log.cc:1196] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/6cfca338808d4e97902429749929bb19
I20260925 15:19:36.221122 16572 ts_tablet_manager.cc:1939] T 6cfca338808d4e97902429749929bb19 P acaa02ee2548458fadd257ac9da60c3e: Deleting consensus metadata
I20260925 15:19:36.223649 15237 catalog_manager.cc:4841] TS acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311): tablet 6cfca338808d4e97902429749929bb19 (table test-table [id=62277fb9655d4e448530d5be31c9b0aa]) successfully deleted
I20260925 15:19:36.225696 16574 tablet_replica.cc:330] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: stopping tablet replica
I20260925 15:19:36.226099 16574 raft_consensus.cc:2229] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:36.226691 16574 raft_consensus.cc:2258] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:36.228945 16573 tablet_replica.cc:330] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: stopping tablet replica
I20260925 15:19:36.231014 16573 raft_consensus.cc:2229] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:36.231526 16573 raft_consensus.cc:2258] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:36.233530 16574 ts_tablet_manager.cc:1905] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:36.238482 16573 ts_tablet_manager.cc:1905] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:36.243974 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.207374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.8.226.125:41525)
0925 15:19:36.207539 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.207546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.216363 (+ 8817us) server_negotiation.cc:407] Connection header received
0925 15:19:36.217474 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.217480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.217559 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.217627 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:36.229094 (+ 11467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.230322 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.232356 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.233062 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.233188 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.236876 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:36.236883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:36.236888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:36.236928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:36.240586 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.241628 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.241633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.241637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.241797 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.242337 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.242342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.242345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.242663 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:36.242875 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.243418 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.243633 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:19:36.245158 16574 ts_tablet_manager.cc:1918] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:36.245326 16574 log.cc:1196] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/6cfca338808d4e97902429749929bb19
I20260925 15:19:36.245961 16574 ts_tablet_manager.cc:1939] T 6cfca338808d4e97902429749929bb19 P 0790d9223cfe4e84b208c1c0c9f5e6d1: Deleting consensus metadata
I20260925 15:19:36.251788 16573 ts_tablet_manager.cc:1918] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:36.251931 16573 log.cc:1196] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/6cfca338808d4e97902429749929bb19
I20260925 15:19:36.252393 16573 ts_tablet_manager.cc:1939] T 6cfca338808d4e97902429749929bb19 P 1ec949ca51d04e958d33aedc426966b8: Deleting consensus metadata
I20260925 15:19:36.261287 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.216558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.8.226.124:43601)
0925 15:19:36.216758 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.216765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.246361 (+ 29596us) server_negotiation.cc:407] Connection header received
0925 15:19:36.246588 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.246593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.246688 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.246759 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:36.248479 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.249522 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.251325 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.251998 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.252091 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.255438 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:36.255444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:36.255449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:36.255483 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:36.258586 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.259293 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.259299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.259303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.259444 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.259812 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.259816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.259820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.260134 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:36.260378 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.260716 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.260922 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:19:36.348865 15235 catalog_manager.cc:4841] TS 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529): tablet 6cfca338808d4e97902429749929bb19 (table test-table [id=62277fb9655d4e448530d5be31c9b0aa]) successfully deleted
I20260925 15:19:36.349237 15235 catalog_manager.cc:4841] TS 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809): tablet 6cfca338808d4e97902429749929bb19 (table test-table [id=62277fb9655d4e448530d5be31c9b0aa]) successfully deleted
I20260925 15:19:36.353230 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15219
I20260925 15:19:36.412734 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.383911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.8.226.124:43601)
0925 15:19:36.385815 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.385821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.395413 (+ 9592us) server_negotiation.cc:407] Connection header received
0925 15:19:36.395877 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.395882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.395967 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.396031 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:36.398218 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.399504 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.401504 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.402132 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.402222 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.405654 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:36.405662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:36.405667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:36.405721 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:36.409292 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.410521 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.410529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.410532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.410705 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.411146 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.411153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.411156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.411535 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:36.411788 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.412167 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.412386 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1777}
I20260925 15:19:36.415414 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:36.399651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.8.226.125:41525)
0925 15:19:36.399862 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:36.399867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:36.400103 (+ 236us) server_negotiation.cc:407] Connection header received
0925 15:19:36.400319 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:36.400325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:36.400419 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:36.400493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:36.402693 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.403713 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.405515 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:36.406145 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:36.406252 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:36.409129 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:36.409135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:36.409140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:36.409186 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:36.412485 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.413545 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.413549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.413552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.413697 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:36.414120 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:36.414124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:36.414126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:36.414458 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:36.414628 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:36.414946 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:19:36.415127 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:19:37.188187 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.187356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:58269)
0925 15:19:37.187804 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.187953 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":217,"threads_started":1}
W20260925 15:19:37.188632 16513 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:33083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.189189 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.188287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:47673)
0925 15:19:37.188831 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.188992 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":324,"threads_started":1}
W20260925 15:19:37.189795 16225 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:33083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.192831 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.192095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:52803)
0925 15:19:37.192518 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.192656 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
W20260925 15:19:37.193143 16372 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:33083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.704139 16586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd5f9acc52c0400fb75f1dd99e5942f4)
I20260925 15:19:37.704463 16587 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd5f9acc52c0400fb75f1dd99e5942f4)
I20260925 15:19:37.704487 16586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:37.704687 16587 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:37.705832 16586 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083), c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125:41525)
I20260925 15:19:37.705855 16587 leader_election.cc:288] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083), 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124:43601)
I20260925 15:19:37.706990 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.705934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60412)
0925 15:19:37.706540 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.706726 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":264,"threads_started":1}
I20260925 15:19:37.707201 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.706016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60410)
0925 15:19:37.706725 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.706912 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":466,"threads_started":1}
W20260925 15:19:37.707293 15833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.709687 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.709220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60422)
0925 15:19:37.709423 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.709512 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 15:19:37.709836 15833 leader_election.cc:334] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
W20260925 15:19:37.710629 16030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.712266 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.711825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60436)
0925 15:19:37.711972 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.712091 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:19:37.712463 16030 leader_election.cc:334] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.716159 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.706810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:41525 (local address 127.0.0.1:59676)
0925 15:19:37.707407 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.707426 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:37.707552 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:37.709524 (+ 1972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:37.709530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:37.709552 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:37.709810 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:37.709824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.711507 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.711511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:37.713569 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:37.713581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.715370 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.715376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:37.715549 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:37.715591 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:19:37.715676 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:19:37.715864 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":23,"thread_start_us":3545,"threads_started":1}
I20260925 15:19:37.716207 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.706648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.8.226.124:43601)
0925 15:19:37.708014 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:37.708022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:37.708664 (+ 642us) server_negotiation.cc:407] Connection header received
0925 15:19:37.708737 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:37.708742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:37.708804 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:37.708905 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:37.710271 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.711400 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.713709 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.714988 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.715107 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:37.715209 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:37.715667 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 15:19:37.715845 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1237,"thread_start_us":1146,"threads_started":1}
I20260925 15:19:37.716403 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.707078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.8.226.125:41525)
0925 15:19:37.709162 (+ 2084us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:37.709170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:37.709193 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:19:37.709261 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:37.709267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:37.709354 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:37.709497 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:37.710075 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.711329 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.713725 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.715158 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.715350 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:37.715524 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:37.715705 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 15:19:37.715877 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":1914,"thread_start_us":1706,"threads_started":1}
I20260925 15:19:37.716430 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.706238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43601 (local address 127.0.0.1:57146)
0925 15:19:37.708398 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.708415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:37.708552 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:37.709193 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:37.709198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:37.709223 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:37.710047 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:37.710061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.711519 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.711523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:37.713552 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:37.713565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.715154 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.715159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:37.715489 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:37.715531 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:19:37.715628 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:19:37.715842 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"mutex_wait_us":29,"thread_start_us":1928,"threads_started":1}
I20260925 15:19:37.716981 15858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" is_pre_election: true
I20260925 15:19:37.717268 15858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9e22b7b2cf44579f9daaca4ccf68a6 in term 1.
I20260925 15:19:37.717196 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" is_pre_election: true
I20260925 15:19:37.717489 16056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01e1cb1827a4f80b47eed915c06d1ed in term 1.
I20260925 15:19:37.717754 16030 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a9e22b7b2cf44579f9daaca4ccf68a6, c01e1cb1827a4f80b47eed915c06d1ed; no voters: dd5f9acc52c0400fb75f1dd99e5942f4
I20260925 15:19:37.717903 15835 leader_election.cc:302] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a9e22b7b2cf44579f9daaca4ccf68a6, c01e1cb1827a4f80b47eed915c06d1ed; no voters: dd5f9acc52c0400fb75f1dd99e5942f4
I20260925 15:19:37.718068 16586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:19:37.718179 16586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd5f9acc52c0400fb75f1dd99e5942f4)
I20260925 15:19:37.718279 16587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:19:37.718297 16586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:19:37.718406 16587 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd5f9acc52c0400fb75f1dd99e5942f4)
I20260925 15:19:37.718516 16587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:19:37.720341 16586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:37.720432 16587 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:37.721113 16586 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 2 election: Requested vote from peers dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083), c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125:41525)
I20260925 15:19:37.721148 16587 leader_election.cc:288] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [CANDIDATE]: Term 2 election: Requested vote from peers dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083), 2a9e22b7b2cf44579f9daaca4ccf68a6 (127.8.226.124:43601)
I20260925 15:19:37.721692 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.721141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60442)
0925 15:19:37.721463 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.721550 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260925 15:19:37.721714 15858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
I20260925 15:19:37.721781 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6"
I20260925 15:19:37.721973 15858 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a9e22b7b2cf44579f9daaca4ccf68a6 in current term 2: Already voted for candidate c01e1cb1827a4f80b47eed915c06d1ed in this term.
I20260925 15:19:37.722043 16056 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c01e1cb1827a4f80b47eed915c06d1ed in current term 2: Already voted for candidate 2a9e22b7b2cf44579f9daaca4ccf68a6 in this term.
I20260925 15:19:37.722384 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.721191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60448)
0925 15:19:37.721561 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.722220 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260925 15:19:37.722668 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.722303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60460)
0925 15:19:37.722477 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.722552 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:19:37.722832 16030 leader_election.cc:334] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.723078 16030 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a9e22b7b2cf44579f9daaca4ccf68a6; no voters: c01e1cb1827a4f80b47eed915c06d1ed, dd5f9acc52c0400fb75f1dd99e5942f4
I20260925 15:19:37.723351 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.722970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60464)
0925 15:19:37.723115 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:37.723205 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:19:37.723390 16586 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 15:19:37.723537 15833 leader_election.cc:334] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:37.723690 15833 leader_election.cc:302] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c01e1cb1827a4f80b47eed915c06d1ed; no voters: 2a9e22b7b2cf44579f9daaca4ccf68a6, dd5f9acc52c0400fb75f1dd99e5942f4
I20260925 15:19:37.723981 16587 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:19:38.190146 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:38.189346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:45655)
0925 15:19:38.189823 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:38.189942 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":229,"threads_started":1}
I20260925 15:19:38.191190 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:38.190446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:55651)
0925 15:19:38.190907 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:38.191000 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260925 15:19:38.194788 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:38.193944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:38587)
0925 15:19:38.194406 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:38.194511 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":270,"threads_started":1}
I20260925 15:19:39.192227 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.191109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:42737)
0925 15:19:39.191632 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.191733 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":303,"threads_started":1}
I20260925 15:19:39.193293 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.192371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:53945)
0925 15:19:39.193046 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.193146 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":426,"threads_started":1}
I20260925 15:19:39.196280 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.195640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:35671)
0925 15:19:39.196044 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.196116 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":222,"threads_started":1}
I20260925 15:19:39.236469 16602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:39.236675 16602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:39.237322 16602 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083), c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125:41525)
I20260925 15:19:39.238365 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.237499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60684)
0925 15:19:39.238032 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.238173 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260925 15:19:39.238453 15858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" is_pre_election: true
I20260925 15:19:39.238694 15858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9e22b7b2cf44579f9daaca4ccf68a6 in term 2.
I20260925 15:19:39.239213 16030 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9e22b7b2cf44579f9daaca4ccf68a6, c01e1cb1827a4f80b47eed915c06d1ed; no voters:
I20260925 15:19:39.239478 16602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 15:19:39.239614 16602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:39.239738 16602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:19:39.239979 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.239607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60690)
0925 15:19:39.239747 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.239809 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:19:39.240144 16030 leader_election.cc:334] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:39.242059 16602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:39.242491 16602 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 3 election: Requested vote from peers dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083), c01e1cb1827a4f80b47eed915c06d1ed (127.8.226.125:41525)
I20260925 15:19:39.242893 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.242572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60692)
0925 15:19:39.242676 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.242778 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:19:39.243044 15858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c01e1cb1827a4f80b47eed915c06d1ed"
I20260925 15:19:39.243245 15858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:19:39.243804 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.243443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60706)
0925 15:19:39.243599 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.243676 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 15:19:39.243949 16030 leader_election.cc:334] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
I20260925 15:19:39.245541 15858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9e22b7b2cf44579f9daaca4ccf68a6 in term 3.
I20260925 15:19:39.245880 16030 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a9e22b7b2cf44579f9daaca4ccf68a6, c01e1cb1827a4f80b47eed915c06d1ed; no voters: dd5f9acc52c0400fb75f1dd99e5942f4
I20260925 15:19:39.246114 16602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:19:39.246452 16602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [term 3 LEADER]: Becoming Leader. State: Replica: 2a9e22b7b2cf44579f9daaca4ccf68a6, State: Running, Role: LEADER
I20260925 15:19:39.246784 16602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } }
I20260925 15:19:39.248209 16605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9e22b7b2cf44579f9daaca4ccf68a6. Latest consensus state: current_term: 3 leader_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:39.248535 16605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:39.249116 16607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:39.252247 16607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:39.253105 16607 catalog_manager.cc:1259] Loaded cluster ID: 4e34ba29bde74c018c248a281a37c778
I20260925 15:19:39.253190 16607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:39.254269 16607 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:39.255024 16607 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Loaded TSK: 0
I20260925 15:19:39.255796 16607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:39.285082 16046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:39.292089 16046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:39.295938 15858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [term 3 FOLLOWER]: Refusing update from remote peer 2a9e22b7b2cf44579f9daaca4ccf68a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 15:19:39.296038 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.295573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60714)
0925 15:19:39.295780 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.295881 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 15:19:39.296557 16605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 15:19:39.297003 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.296595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60724)
0925 15:19:39.296774 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.296849 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:19:39.297183 16030 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 -> Peer dd5f9acc52c0400fb75f1dd99e5942f4 (127.8.226.126:33083): Couldn't send request to peer dd5f9acc52c0400fb75f1dd99e5942f4. Status: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:19:39.299260 16609 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9e22b7b2cf44579f9daaca4ccf68a6. Latest consensus state: current_term: 3 leader_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:39.299521 16609 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:39.300782 16602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:39.301144 16602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:39.301762 16609 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dd5f9acc52c0400fb75f1dd99e5942f4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33083 } } peers { permanent_uuid: "c01e1cb1827a4f80b47eed915c06d1ed" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 41525 } attrs { promote: false } } peers { permanent_uuid: "2a9e22b7b2cf44579f9daaca4ccf68a6" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43601 } attrs { promote: false } } }
I20260925 15:19:39.302000 16609 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01e1cb1827a4f80b47eed915c06d1ed [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:39.340152 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.308771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:39809 (local address 127.0.0.1:35112)
0925 15:19:39.309093 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.309116 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:39.309250 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:39.333697 (+ 24447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:39.333702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:39.333728 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:39.334040 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.334049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.335778 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.335783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.338044 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.338058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.339489 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.339494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.339651 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.339683 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:19:39.339736 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:19:39.339853 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":70}
I20260925 15:19:39.340673 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.310144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:33529 (local address 127.0.0.1:43152)
0925 15:19:39.332221 (+ 22077us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.332241 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:39.332333 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:39.333858 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:39.333862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:39.333890 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:39.334555 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.334568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.335895 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.335899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.338421 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.338433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.339889 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.339894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.340065 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.340093 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:19:39.340159 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:19:39.340264 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21915,"mutex_wait_us":336,"thread_start_us":22188,"threads_started":1}
I20260925 15:19:39.341019 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.308886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.8.226.66:39809)
0925 15:19:39.333388 (+ 24502us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.333395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.333414 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:19:39.333480 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.333485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.333553 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.333654 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:39.334212 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.335591 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.338206 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.339310 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.340395 (+ 1085us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.340510 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.340576 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.340701 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24373,"thread_start_us":24255,"threads_started":1}
I20260925 15:19:39.341651 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.309994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.8.226.67:33529)
0925 15:19:39.332738 (+ 22744us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.332744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.332764 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:19:39.332835 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.332842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.332923 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.333948 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:39.334745 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.335762 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.338584 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.339689 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.341056 (+ 1367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.341171 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.341219 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.341357 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22615,"thread_start_us":26084,"threads_started":1}
I20260925 15:19:39.341794 16306 tablet_service.cc:1467] Processing CreateTablet for tablet f3d416a41af6401ba422073785058b0e (DEFAULT_TABLE table=test-table [id=ea00a692e58e46ab887ee42519b82d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:39.342233 16306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d416a41af6401ba422073785058b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:39.342262 16447 tablet_service.cc:1467] Processing CreateTablet for tablet f3d416a41af6401ba422073785058b0e (DEFAULT_TABLE table=test-table [id=ea00a692e58e46ab887ee42519b82d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:39.342655 16447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d416a41af6401ba422073785058b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:39.346674 16620 tablet_bootstrap.cc:492] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Bootstrap starting.
I20260925 15:19:39.346712 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.309390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43311 (local address 127.0.0.1:53198)
0925 15:19:39.331620 (+ 22230us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.331636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:39.331791 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:39.332525 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:39.332529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:39.332554 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:39.332881 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.332894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.334303 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.334308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.338430 (+ 4122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.338437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.340070 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.340075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.340513 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.340563 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:19:39.340647 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:19:39.341033 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22134,"mutex_wait_us":53,"thread_start_us":26375,"threads_started":1}
I20260925 15:19:39.347671 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.309703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.8.226.65:43311)
0925 15:19:39.325939 (+ 16236us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.325948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.331731 (+ 5783us) server_negotiation.cc:407] Connection header received
0925 15:19:39.331993 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.331998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.332082 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.332173 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:39.333112 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.334151 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.338584 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.339706 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.339816 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.339922 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.347244 (+ 7322us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.347376 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16060,"thread_start_us":16251,"threads_started":1}
I20260925 15:19:39.348300 16159 tablet_service.cc:1467] Processing CreateTablet for tablet f3d416a41af6401ba422073785058b0e (DEFAULT_TABLE table=test-table [id=ea00a692e58e46ab887ee42519b82d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:39.348397 16621 tablet_bootstrap.cc:492] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Bootstrap starting.
I20260925 15:19:39.348685 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d416a41af6401ba422073785058b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:39.348760 16620 tablet_bootstrap.cc:654] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:39.350133 16621 tablet_bootstrap.cc:654] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:39.359747 16620 tablet_bootstrap.cc:492] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: No bootstrap required, opened a new log
I20260925 15:19:39.359896 16620 ts_tablet_manager.cc:1397] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260925 15:19:39.360342 16621 tablet_bootstrap.cc:492] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: No bootstrap required, opened a new log
I20260925 15:19:39.360486 16621 ts_tablet_manager.cc:1397] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260925 15:19:39.360692 16620 raft_consensus.cc:348] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.360927 16620 raft_consensus.cc:374] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:39.361063 16620 raft_consensus.cc:729] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0790d9223cfe4e84b208c1c0c9f5e6d1, State: Initialized, Role: FOLLOWER
I20260925 15:19:39.361104 16624 tablet_bootstrap.cc:492] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Bootstrap starting.
I20260925 15:19:39.361306 16621 raft_consensus.cc:348] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.361297 16620 consensus_queue.cc:260] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.361514 16621 raft_consensus.cc:374] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:39.361605 16621 raft_consensus.cc:729] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec949ca51d04e958d33aedc426966b8, State: Initialized, Role: FOLLOWER
I20260925 15:19:39.361914 16621 consensus_queue.cc:260] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.362751 16624 tablet_bootstrap.cc:654] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:39.362870 16621 ts_tablet_manager.cc:1428] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:39.363626 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.363179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:34839)
0925 15:19:39.363397 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.363487 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 15:19:39.365633 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.365214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:58703)
0925 15:19:39.365422 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.365504 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 15:19:39.366604 16512 heartbeater.cc:503] Master 127.8.226.124:43601 was elected leader, sending a full tablet report...
I20260925 15:19:39.367208 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.366683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:44953)
0925 15:19:39.366995 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.367077 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260925 15:19:39.367930 16620 ts_tablet_manager.cc:1428] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20260925 15:19:39.368317 16371 heartbeater.cc:503] Master 127.8.226.124:43601 was elected leader, sending a full tablet report...
I20260925 15:19:39.368811 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.368367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:60479)
0925 15:19:39.368565 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.368644 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 15:19:39.370996 16624 tablet_bootstrap.cc:492] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: No bootstrap required, opened a new log
I20260925 15:19:39.371141 16624 ts_tablet_manager.cc:1397] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260925 15:19:39.371778 16624 raft_consensus.cc:348] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.371963 16624 raft_consensus.cc:374] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:39.372094 16624 raft_consensus.cc:729] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acaa02ee2548458fadd257ac9da60c3e, State: Initialized, Role: FOLLOWER
I20260925 15:19:39.372330 16624 consensus_queue.cc:260] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.373067 16624 ts_tablet_manager.cc:1428] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:39.373715 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.373288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:35169)
0925 15:19:39.373459 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.373553 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:19:39.374616 16628 raft_consensus.cc:482] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:39.374795 16224 heartbeater.cc:503] Master 127.8.226.124:43601 was elected leader, sending a full tablet report...
I20260925 15:19:39.374866 16628 raft_consensus.cc:504] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.375630 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.375278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:57739)
0925 15:19:39.375432 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.375503 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:19:39.377580 16628 leader_election.cc:288] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809), 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529)
I20260925 15:19:39.383668 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.377847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:56067 (local address 127.8.226.66:39809)
0925 15:19:39.378021 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.378027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.378076 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:19:39.378150 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.378154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.378238 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.378323 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:39.378978 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.379810 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.381989 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.382798 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.382959 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.383024 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.383299 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.383425 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:19:39.383733 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.377134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:39809 (local address 127.8.226.65:56067)
0925 15:19:39.377260 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.377276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:39.377378 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:39.378478 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:39.378483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:39.378508 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:39.378817 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.378831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.379937 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.379944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.381813 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.381823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.382968 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.382973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.383156 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.383178 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:19:39.383279 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:19:39.383404 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":3}
I20260925 15:19:39.384402 16326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d416a41af6401ba422073785058b0e" candidate_uuid: "acaa02ee2548458fadd257ac9da60c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" is_pre_election: true
I20260925 15:19:39.384742 16326 raft_consensus.cc:2454] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acaa02ee2548458fadd257ac9da60c3e in term 0.
I20260925 15:19:39.385250 16112 leader_election.cc:302] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0790d9223cfe4e84b208c1c0c9f5e6d1, acaa02ee2548458fadd257ac9da60c3e; no voters:
I20260925 15:19:39.385581 16628 raft_consensus.cc:2781] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:39.385722 16628 raft_consensus.cc:482] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:39.385831 16628 raft_consensus.cc:3037] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:39.387190 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.378153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:57451 (local address 127.8.226.67:33529)
0925 15:19:39.378325 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.378330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.380510 (+ 2180us) server_negotiation.cc:407] Connection header received
0925 15:19:39.380661 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.380666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.380729 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.380810 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:39.381602 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.382579 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.385004 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.386136 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.386238 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.386307 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.386825 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.386984 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:19:39.387223 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.377771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:33529 (local address 127.8.226.65:57451)
0925 15:19:39.380416 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.380433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:39.380551 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:39.380951 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:39.380956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:39.381002 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:39.381403 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.381450 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.382724 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.382729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.384828 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:39.384840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.386481 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.386487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:39.386667 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.386697 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:19:39.386781 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:19:39.386913 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2510,"thread_start_us":2295,"threads_started":1}
I20260925 15:19:39.387732 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d416a41af6401ba422073785058b0e" candidate_uuid: "acaa02ee2548458fadd257ac9da60c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec949ca51d04e958d33aedc426966b8" is_pre_election: true
I20260925 15:19:39.387944 16467 raft_consensus.cc:2454] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acaa02ee2548458fadd257ac9da60c3e in term 0.
I20260925 15:19:39.388504 16628 raft_consensus.cc:504] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.389140 16628 leader_election.cc:288] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [CANDIDATE]: Term 1 election: Requested vote from peers 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809), 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529)
I20260925 15:19:39.389521 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d416a41af6401ba422073785058b0e" candidate_uuid: "acaa02ee2548458fadd257ac9da60c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec949ca51d04e958d33aedc426966b8"
I20260925 15:19:39.389710 16467 raft_consensus.cc:3037] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:39.389762 16326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d416a41af6401ba422073785058b0e" candidate_uuid: "acaa02ee2548458fadd257ac9da60c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1"
I20260925 15:19:39.389954 16326 raft_consensus.cc:3037] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:39.391729 16467 raft_consensus.cc:2454] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acaa02ee2548458fadd257ac9da60c3e in term 1.
I20260925 15:19:39.392011 16326 raft_consensus.cc:2454] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acaa02ee2548458fadd257ac9da60c3e in term 1.
I20260925 15:19:39.392046 16112 leader_election.cc:302] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec949ca51d04e958d33aedc426966b8, acaa02ee2548458fadd257ac9da60c3e; no voters:
I20260925 15:19:39.392355 16628 raft_consensus.cc:2781] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:39.392766 16628 raft_consensus.cc:686] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 1 LEADER]: Becoming Leader. State: Replica: acaa02ee2548458fadd257ac9da60c3e, State: Running, Role: LEADER
I20260925 15:19:39.393179 16628 consensus_queue.cc:237] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } }
I20260925 15:19:39.393693 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.393361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:54951)
0925 15:19:39.393497 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.393589 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:19:39.396445 16045 catalog_manager.cc:5495] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e reported cstate change: term changed from 0 to 1, leader changed from <none> to acaa02ee2548458fadd257ac9da60c3e (127.8.226.65). New cstate: current_term: 1 leader_uuid: "acaa02ee2548458fadd257ac9da60c3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acaa02ee2548458fadd257ac9da60c3e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:39.424414 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.419648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.8.226.65:43311)
0925 15:19:39.419814 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.419818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.419920 (+ 102us) server_negotiation.cc:407] Connection header received
0925 15:19:39.420091 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.420095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.420160 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.420274 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:39.420575 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.421323 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.422483 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.422878 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.422958 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.423297 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:39.423858 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:39.423998 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.424106 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.424162 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:19:39.428059 16467 raft_consensus.cc:1261] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Refusing update from remote peer acaa02ee2548458fadd257ac9da60c3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:39.428303 16326 raft_consensus.cc:1261] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Refusing update from remote peer acaa02ee2548458fadd257ac9da60c3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:39.428668 16628 consensus_queue.cc:1035] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec949ca51d04e958d33aedc426966b8" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:39.429026 16630 consensus_queue.cc:1035] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0790d9223cfe4e84b208c1c0c9f5e6d1" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:39.431429 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.430952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:42385)
0925 15:19:39.431181 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.431268 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 15:19:39.432651 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.432130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:51893)
0925 15:19:39.432372 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.432474 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260925 15:19:39.435238 16633 mvcc.cc:205] Tried to move back new op lower bound from 7333271877331136512 to 7333271877198577664. Current Snapshot: MvccSnapshot[applied={T|T < 7333271877331136512}]
I20260925 15:19:39.437093 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.436477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.66:42539)
0925 15:19:39.436753 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.436868 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 15:19:39.438648 16515 tablet.cc:2369] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:39.439889 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.439238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.67:52939)
0925 15:19:39.439587 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.439699 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260925 15:19:39.445989 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.445551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:54109)
0925 15:19:39.445756 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.445838 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:19:39.446902 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.446528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.8.226.65:51865)
0925 15:19:39.446693 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.446774 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:19:39.456961 16045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:39.457242 16045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:39.462543 16045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9e22b7b2cf44579f9daaca4ccf68a6: Sending DeleteTablet for 3 replicas of tablet f3d416a41af6401ba422073785058b0e
I20260925 15:19:39.463389 16306 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d416a41af6401ba422073785058b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:39 UTC) from {username='oryx', principal='oryx/127.8.226.124@KRBTEST.COM'} at 127.0.0.1:35112
I20260925 15:19:39.463717 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d416a41af6401ba422073785058b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:39 UTC) from {username='oryx', principal='oryx/127.8.226.124@KRBTEST.COM'} at 127.0.0.1:43152
I20260925 15:19:39.463778 16159 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d416a41af6401ba422073785058b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:39 UTC) from {username='oryx', principal='oryx/127.8.226.124@KRBTEST.COM'} at 127.0.0.1:53198
I20260925 15:19:39.463977 16644 tablet_replica.cc:330] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: stopping tablet replica
I20260925 15:19:39.464160 16644 raft_consensus.cc:2229] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:39.464432 16644 raft_consensus.cc:2258] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:39.465407 16644 ts_tablet_manager.cc:1905] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:39.466138 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 16018
I20260925 15:19:39.471657 16644 ts_tablet_manager.cc:1918] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:39.471817 16644 log.cc:1196] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/f3d416a41af6401ba422073785058b0e
I20260925 15:19:39.472293 16644 ts_tablet_manager.cc:1939] T f3d416a41af6401ba422073785058b0e P 0790d9223cfe4e84b208c1c0c9f5e6d1: Deleting consensus metadata
I20260925 15:19:39.473338 16030 catalog_manager.cc:4841] TS 0790d9223cfe4e84b208c1c0c9f5e6d1 (127.8.226.66:39809): tablet f3d416a41af6401ba422073785058b0e (table test-table [id=ea00a692e58e46ab887ee42519b82d40]) successfully deleted
I20260925 15:19:39.477880 16646 tablet_replica.cc:330] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: stopping tablet replica
I20260925 15:19:39.478108 16646 raft_consensus.cc:2229] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:39.478716 16646 raft_consensus.cc:2258] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:39.479318 16645 tablet_replica.cc:330] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: stopping tablet replica
I20260925 15:19:39.479501 16645 raft_consensus.cc:2229] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:39.479803 16645 raft_consensus.cc:2258] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:39.479822 16646 ts_tablet_manager.cc:1905] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:39.481051 16645 ts_tablet_manager.cc:1905] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:39.486910 16646 ts_tablet_manager.cc:1918] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:39.487046 16646 log.cc:1196] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/f3d416a41af6401ba422073785058b0e
I20260925 15:19:39.487211 16645 ts_tablet_manager.cc:1918] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:39.487329 16645 log.cc:1196] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/f3d416a41af6401ba422073785058b0e
I20260925 15:19:39.487522 16646 ts_tablet_manager.cc:1939] T f3d416a41af6401ba422073785058b0e P acaa02ee2548458fadd257ac9da60c3e: Deleting consensus metadata
I20260925 15:19:39.487738 16645 ts_tablet_manager.cc:1939] T f3d416a41af6401ba422073785058b0e P 1ec949ca51d04e958d33aedc426966b8: Deleting consensus metadata
I20260925 15:19:39.488584 16030 catalog_manager.cc:4841] TS 1ec949ca51d04e958d33aedc426966b8 (127.8.226.67:33529): tablet f3d416a41af6401ba422073785058b0e (table test-table [id=ea00a692e58e46ab887ee42519b82d40]) successfully deleted
I20260925 15:19:39.488583 16032 catalog_manager.cc:4841] TS acaa02ee2548458fadd257ac9da60c3e (127.8.226.65:43311): tablet f3d416a41af6401ba422073785058b0e (table test-table [id=ea00a692e58e46ab887ee42519b82d40]) successfully deleted
I20260925 15:19:39.493894 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.468969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.8.226.65:43311)
0925 15:19:39.469174 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.469183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.477379 (+ 8196us) server_negotiation.cc:407] Connection header received
0925 15:19:39.477542 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.477552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.477645 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.477768 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:39.479809 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.481066 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.483206 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.483822 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.483911 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.487372 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:39.487386 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:39.487405 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:39.487442 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:39.491265 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:39.491880 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:39.491889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:39.491893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:39.492076 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:39.492398 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:39.492403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:39.492406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:39.492826 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:39.493155 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.493333 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.493526 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:19:39.494522 16219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:39.596504 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 16018
I20260925 15:19:39.614881 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 16236
I20260925 15:19:39.627542 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.615566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.8.226.66:39809)
0925 15:19:39.615767 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.615771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.615826 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:19:39.615975 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.615978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.616035 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.616110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:39.617460 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.618168 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.619166 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.619503 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.619583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.622463 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:39.622478 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:39.622493 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:39.622526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:39.625373 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:39.625940 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:39.625946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:39.625948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:39.626073 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:39.626345 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:39.626348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:39.626350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:39.626648 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:39.626859 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.627025 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.627182 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:19:39.628219 16366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:39.742520 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 16236
I20260925 15:19:39.762998 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 16384
I20260925 15:19:39.775586 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.763828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.8.226.67:33529)
0925 15:19:39.763992 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.764000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.764079 (+ 79us) server_negotiation.cc:407] Connection header received
0925 15:19:39.764289 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.764293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.764355 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.764457 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:39.765803 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.766543 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.767778 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:39.768210 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:39.768336 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:39.770577 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:39.770602 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:39.770617 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:39.770656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:39.773118 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:39.773732 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:39.773741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:39.773745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:39.773867 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:39.774212 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:39.774217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:39.774220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:39.774534 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:39.774822 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.775081 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.775282 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:19:39.776180 16507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:39.787039 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.786681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60740)
0925 15:19:39.786825 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.786909 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:19:39.787925 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.787600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33083 (local address 127.0.0.1:60746)
0925 15:19:39.787744 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:39.787813 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:19:39.866073 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 16384
I20260925 15:19:39.882694 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15712
I20260925 15:19:39.883754 15879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:39.993912 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15712
I20260925 15:19:40.013710 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 15811
I20260925 15:19:40.014696 16076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:40.130235 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 15811
2026-09-25T15:19:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10431 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:40 dist-test-slave-8vw7 krb5kdc[16655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:40 dist-test-slave-8vw7 krb5kdc[16655](info): set up 2 sockets
Sep 25 15:19:40 dist-test-slave-8vw7 krb5kdc[16655](info): commencing operation
krb5kdc: starting...
W20260925 15:19:41.165619 9097 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:41 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab.
Entry for principal oryx/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab.
I20260925 15:19:41.364027 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:33777
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab
--principal=oryx/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:41.663759 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:41.664093 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:41.664251 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:41.670651 16668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:41.670776 16668 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:19:41.670867 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:41.670982 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:41.671070 16668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:41.671154 16668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:41.679489 16668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:33777
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.126.keytab
--principal=oryx/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:41.680549 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:41.681811 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:41.692794 16676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:41.692838 16674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:41.692831 16673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:41.693539 16668 server_base.cc:1034] running on GCE node
I20260925 15:19:41.694192 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:41.695356 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:41.696578 16668 hybrid_clock.cc:648] HybridClock initialized: now 1790349581696514 us; error 42 us; skew 500 ppm
I20260925 15:19:41.696873 16668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:41 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349581, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:41.699685 16668 init.cc:377] Logged in from keytab as oryx/127.8.226.126@KRBTEST.COM (short username oryx)
I20260925 15:19:41.701017 16668 webserver.cc:460] Webserver started at http://127.8.226.126:35735/ using document root <none> and password file <none>
I20260925 15:19:41.701660 16668 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:41.701786 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:41.702095 16668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:41.704715 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "e84da461e39a4670964776f35a6f2fbc"
format_stamp: "Formatted at 2026-09-25 15:19:41 on dist-test-slave-8vw7"
server_key: "b79123a40746e4552e5c83b341232c5a"
server_key_iv: "06f1f1c61e8ed85f09e07d439d150873"
server_key_version: "encryptionkey@0"
I20260925 15:19:41.705476 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "e84da461e39a4670964776f35a6f2fbc"
format_stamp: "Formatted at 2026-09-25 15:19:41 on dist-test-slave-8vw7"
server_key: "b79123a40746e4552e5c83b341232c5a"
server_key_iv: "06f1f1c61e8ed85f09e07d439d150873"
server_key_version: "encryptionkey@0"
I20260925 15:19:41.710124 16668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:19:41.713218 16683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:41.714443 16668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:41.714614 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "e84da461e39a4670964776f35a6f2fbc"
format_stamp: "Formatted at 2026-09-25 15:19:41 on dist-test-slave-8vw7"
server_key: "b79123a40746e4552e5c83b341232c5a"
server_key_iv: "06f1f1c61e8ed85f09e07d439d150873"
server_key_version: "encryptionkey@0"
I20260925 15:19:41.714818 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:41.736312 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:41.739758 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:41.740006 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:41.760624 16668 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:33777
I20260925 15:19:41.760632 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:33777 every 8 connection(s)
I20260925 15:19:41.761941 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:41.762354 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 16668
I20260925 15:19:41.762588 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:41.763404 9097 external_mini_cluster.cc:1428] Setting key 9dbb098e2d6cce7f0476a9996b090670
I20260925 15:19:41.766273 16745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:19:41 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:41.778501 16745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Bootstrap starting.
I20260925 15:19:41.782241 16745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:41.783169 16745 log.cc:824] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:41.785581 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:41.765296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.8.226.126:33777)
0925 15:19:41.766029 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:41.766039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:41.766076 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:19:41.767666 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:41.767687 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:41.768029 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:41.768333 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:41.770024 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:41.771345 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:41.773061 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:41.773678 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:41.773806 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:41.778187 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:41.778198 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:41.778215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:41.778264 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:41.781823 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:41.782484 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:41.782490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:41.782495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:41.782653 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:41.783014 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:41.783018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:41.783020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:41.783541 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:41.783841 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:41.784273 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 15:19:41.784646 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":339,"threads_started":1}
I20260925 15:19:41.785892 16745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: No bootstrap required, opened a new log
I20260925 15:19:41.792514 16745 raft_consensus.cc:348] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84da461e39a4670964776f35a6f2fbc" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33777 } }
I20260925 15:19:41.792806 16745 raft_consensus.cc:374] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:41.792906 16745 raft_consensus.cc:729] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84da461e39a4670964776f35a6f2fbc, State: Initialized, Role: FOLLOWER
I20260925 15:19:41.793413 16745 consensus_queue.cc:260] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84da461e39a4670964776f35a6f2fbc" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33777 } }
I20260925 15:19:41.793653 16745 raft_consensus.cc:388] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:41.793840 16745 raft_consensus.cc:482] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:41.794027 16745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:41.795886 16745 raft_consensus.cc:504] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84da461e39a4670964776f35a6f2fbc" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33777 } }
I20260925 15:19:41.796259 16745 leader_election.cc:302] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e84da461e39a4670964776f35a6f2fbc; no voters:
I20260925 15:19:41.796694 16745 leader_election.cc:288] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:41.796847 16750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:41.797502 16750 raft_consensus.cc:686] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [term 1 LEADER]: Becoming Leader. State: Replica: e84da461e39a4670964776f35a6f2fbc, State: Running, Role: LEADER
I20260925 15:19:41.797973 16750 consensus_queue.cc:237] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84da461e39a4670964776f35a6f2fbc" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33777 } }
I20260925 15:19:41.798532 16745 sys_catalog.cc:564] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:41.801218 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e84da461e39a4670964776f35a6f2fbc. Latest consensus state: current_term: 1 leader_uuid: "e84da461e39a4670964776f35a6f2fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84da461e39a4670964776f35a6f2fbc" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33777 } } }
I20260925 15:19:41.801692 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:41.801306 16751 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e84da461e39a4670964776f35a6f2fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84da461e39a4670964776f35a6f2fbc" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 33777 } } }
I20260925 15:19:41.801988 16751 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:41.803092 16759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:41.807909 16759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:41.815270 16759 catalog_manager.cc:1347] Generated new cluster ID: f56faba18ba94253b942358a83350061
I20260925 15:19:41.815361 16759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:41.826812 16759 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:41.828003 16759 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:41.842983 16759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Generated new TSK 0
I20260925 15:19:41.843762 16759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab.
Entry for principal oryx/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab.
I20260925 15:19:41.894043 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33777
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:42.210148 16771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:42.210390 16771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:42.210479 16771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:42.216852 16771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:42.216984 16771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:42.217060 16771 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:19:42.217131 16771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:42.225561 16771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:33777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:42.226593 16771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:42.227833 16771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:42.239832 16776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:42.239813 16777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:42.239825 16779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:42.240763 16771 server_base.cc:1034] running on GCE node
I20260925 15:19:42.241354 16771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:42.242098 16771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:42.243328 16771 hybrid_clock.cc:648] HybridClock initialized: now 1790349582243272 us; error 32 us; skew 500 ppm
I20260925 15:19:42.243602 16771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:42 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349582, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:42.246363 16771 init.cc:377] Logged in from keytab as oryx/127.8.226.65@KRBTEST.COM (short username oryx)
I20260925 15:19:42.247643 16771 webserver.cc:460] Webserver started at http://127.8.226.65:32793/ using document root <none> and password file <none>
I20260925 15:19:42.248274 16771 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:42.248394 16771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:42.248672 16771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:42.251390 16771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "d38da62da3ec4492bb1ada9e416387d3"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-8vw7"
server_key: "cdd140c954ec85046541e8cc95471ca3"
server_key_iv: "c7300403172a8e8e349c8214e1c12aac"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.252144 16771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "d38da62da3ec4492bb1ada9e416387d3"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-8vw7"
server_key: "cdd140c954ec85046541e8cc95471ca3"
server_key_iv: "c7300403172a8e8e349c8214e1c12aac"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.256656 16771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:19:42.259675 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:42.260808 16771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:42.261003 16771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "d38da62da3ec4492bb1ada9e416387d3"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-8vw7"
server_key: "cdd140c954ec85046541e8cc95471ca3"
server_key_iv: "c7300403172a8e8e349c8214e1c12aac"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.261207 16771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:42.280910 16771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:42.284502 16771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:42.284790 16771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:42.287257 16771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:42.287354 16771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:42.287505 16771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:42.287598 16771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 15:19:42 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349582, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:42.307298 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.287710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:36639 (local address 127.8.226.126:33777)
0925 15:19:42.289079 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:42.289088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:42.289655 (+ 567us) server_negotiation.cc:407] Connection header received
0925 15:19:42.291583 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:42.291589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:42.291667 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:42.291797 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:42.293730 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.294917 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.297290 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.297643 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.297771 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.300889 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:42.300894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:42.300897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:42.300927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:42.303862 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.304596 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.304601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.304605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.304738 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.305183 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.305188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.305191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.305524 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:42.305993 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:42.306613 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 15:19:42.306866 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1074,"threads_started":1}
I20260925 15:19:42.308843 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.288392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.8.226.65:36639)
0925 15:19:42.289479 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:42.289520 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:42.291276 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:42.291992 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:42.291999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:42.292668 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:42.293473 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.293502 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.295082 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.295090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.297160 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.297170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.297860 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.297868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.298053 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.298601 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:42.298623 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:42.300641 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:42.304005 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.304013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.304026 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.304476 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.304836 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.304841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.304844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.305072 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.305668 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:42.305674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:42.306099 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 15:19:42.306497 (+ 398us) client_negotiation.cc:240] Negotiation successful
0925 15:19:42.307092 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":644,"threads_started":1}
I20260925 15:19:42.329690 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.314619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.8.226.65:51027)
0925 15:19:42.315401 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:42.315414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:42.315565 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:42.315889 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:42.315894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:42.316299 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:42.316541 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.316548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.317633 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.317639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.318716 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.318728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.320029 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.320041 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.320387 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.321166 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:42.321192 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:42.322713 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:42.326395 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.326403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.326408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.326973 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.327464 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.327476 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.327482 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.327631 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.328218 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:42.328222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:42.328476 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 15:19:42.328686 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 15:19:42.329038 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":493,"threads_started":1}
I20260925 15:19:42.330371 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.314525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:51027 (local address 127.8.226.126:33777)
0925 15:19:42.314704 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:42.314709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:42.315508 (+ 799us) server_negotiation.cc:407] Connection header received
0925 15:19:42.315665 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:42.315670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:42.315742 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:42.315852 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:42.316665 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.317522 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.318857 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.319622 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.319761 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.322959 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:42.322968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:42.322975 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:42.323015 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:42.326231 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.327122 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.327129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.327135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.327316 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.327759 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.327766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.327770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.328118 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:42.328260 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:42.329825 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0925 15:19:42.330081 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:19:42.338589 16771 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:37857
I20260925 15:19:42.338608 16906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:37857 every 8 connection(s)
I20260925 15:19:42.339684 16771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:42.341506 16907 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33777
I20260925 15:19:42.341729 16907 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:42.342394 16907 heartbeater.cc:511] Master 127.8.226.126:33777 requested a full tablet report, sending...
I20260925 15:19:42.344322 16700 ts_manager.cc:194] Registered new tserver with Master: d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857)
I20260925 15:19:42.345311 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 16771
I20260925 15:19:42.345523 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:42.346110 16700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:36639
I20260925 15:19:42.346257 9097 external_mini_cluster.cc:1428] Setting key e7fb6ae37ec6af2e4f6bc2e6bf6d3689
WARNING: no policy specified for oryx/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab.
Entry for principal oryx/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab.
I20260925 15:19:42.387938 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33777
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:42.687743 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:42.688061 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:42.688247 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:42.695403 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:42.695551 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:42.695680 16911 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:19:42.695822 16911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:42.705205 16911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:33777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:42.706461 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:42.707860 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:42.721515 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:42.721578 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:42.721602 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:42.722317 16911 server_base.cc:1034] running on GCE node
I20260925 15:19:42.722838 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:42.723655 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:42.724865 16911 hybrid_clock.cc:648] HybridClock initialized: now 1790349582724799 us; error 38 us; skew 500 ppm
I20260925 15:19:42.725199 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:42 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349582, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:42.728153 16911 init.cc:377] Logged in from keytab as oryx/127.8.226.66@KRBTEST.COM (short username oryx)
I20260925 15:19:42.729887 16911 webserver.cc:460] Webserver started at http://127.8.226.66:39471/ using document root <none> and password file <none>
I20260925 15:19:42.730618 16911 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:42.730755 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:42.731098 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:42.733971 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "57ce821999cb4f4aa8bf48d6024d6ad7"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-8vw7"
server_key: "9b858a383475e2062d4044e3b1220429"
server_key_iv: "7c6a3ffe95d03224e28f925cf3da7040"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.734761 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "57ce821999cb4f4aa8bf48d6024d6ad7"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-8vw7"
server_key: "9b858a383475e2062d4044e3b1220429"
server_key_iv: "7c6a3ffe95d03224e28f925cf3da7040"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.739755 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:19:42.742990 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:42.744179 16911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:42.744342 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "57ce821999cb4f4aa8bf48d6024d6ad7"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-8vw7"
server_key: "9b858a383475e2062d4044e3b1220429"
server_key_iv: "7c6a3ffe95d03224e28f925cf3da7040"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.744546 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:42.756861 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:42.760439 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:42.760694 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:42.763378 16911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:42.763509 16911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:42.763688 16911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:42.763761 16911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:19:42 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349582, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:42.787225 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.764013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:35163 (local address 127.8.226.126:33777)
0925 15:19:42.764568 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:42.764574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:42.765637 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 15:19:42.768355 (+ 2718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:42.768361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:42.768437 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:42.768508 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:42.770643 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.771774 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.774489 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.775139 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.775260 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.779745 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:42.779751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:42.779756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:42.779793 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:42.783255 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.784183 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.784188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.784192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.784362 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.784955 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.784987 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.784993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.785360 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:42.785526 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:42.786449 (+ 923us) server_negotiation.cc:299] Negotiation successful
0925 15:19:42.786753 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441}
I20260925 15:19:42.788525 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.764630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.8.226.66:35163)
0925 15:19:42.765487 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:42.765534 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:42.768040 (+ 2506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:42.769002 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:42.769021 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:42.769739 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:42.770459 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.770482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.771998 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.772003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.774324 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.774338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.775540 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.775550 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.775917 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.776654 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:42.776703 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:42.779483 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:42.783397 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.783405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.783424 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.784055 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.784486 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.784497 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.784503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.784803 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.785506 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:42.785513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:42.785978 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 15:19:42.786288 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 15:19:42.786942 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
I20260925 15:19:42.807178 16911 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:35325
I20260925 15:19:42.807195 17045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:35325 every 8 connection(s)
I20260925 15:19:42.808344 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:42.810787 17046 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33777
I20260925 15:19:42.811019 17046 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:42.811173 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.795640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:52373 (local address 127.8.226.126:33777)
0925 15:19:42.795817 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:42.795822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:42.796625 (+ 803us) server_negotiation.cc:407] Connection header received
0925 15:19:42.796862 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:42.796867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:42.796931 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:42.797057 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:42.798684 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.799754 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.801672 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.802411 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.802544 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.805431 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:42.805436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:42.805439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:42.805485 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:42.808087 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.808880 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.808889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.808895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.809105 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:42.809526 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:42.809532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:42.809538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:42.809964 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:42.810152 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:42.810594 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 15:19:42.810819 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:19:42.811291 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.795824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.8.226.66:52373)
0925 15:19:42.796508 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:42.796531 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:42.796678 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:42.797232 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:42.797237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:42.797917 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:42.798496 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.798515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.799896 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.799906 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.801429 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:42.801445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.802818 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.802844 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:42.803114 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.803866 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:42.803895 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:42.805266 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:42.808196 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.808200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.808202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.808672 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.809230 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:42.809235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:42.809238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:42.809395 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:42.810119 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:42.810124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:42.810303 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:19:42.810470 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:19:42.810788 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":390,"threads_started":1}
I20260925 15:19:42.811957 17046 heartbeater.cc:511] Master 127.8.226.126:33777 requested a full tablet report, sending...
I20260925 15:19:42.813316 16700 ts_manager.cc:194] Registered new tserver with Master: 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325)
I20260925 15:19:42.814043 16700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:35163
I20260925 15:19:42.816001 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 16911
I20260925 15:19:42.816170 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:42.816709 9097 external_mini_cluster.cc:1428] Setting key b1afa0121e5fc82c076a6ec99b082e03
WARNING: no policy specified for oryx/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab.
Entry for principal oryx/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab.
I20260925 15:19:42.858011 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:33777
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:43.161518 17050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:43.161772 17050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:43.161865 17050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:43.168632 17050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:43.168718 17050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:43.168792 17050 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:19:43.168862 17050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:43.177727 17050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.226.126:33777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:43.178750 17050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:43.179916 17050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:43.192333 17055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:43.192261 17058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:43.192348 17056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:43.193279 17050 server_base.cc:1034] running on GCE node
I20260925 15:19:43.193847 17050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:43.194646 17050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:43.195883 17050 hybrid_clock.cc:648] HybridClock initialized: now 1790349583195822 us; error 60 us; skew 500 ppm
I20260925 15:19:43.196156 17050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:43 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349583, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:43.199129 17050 init.cc:377] Logged in from keytab as oryx/127.8.226.67@KRBTEST.COM (short username oryx)
I20260925 15:19:43.200409 17050 webserver.cc:460] Webserver started at http://127.8.226.67:45383/ using document root <none> and password file <none>
I20260925 15:19:43.201102 17050 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:43.201222 17050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:43.201537 17050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:43.204319 17050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "f0147c8ca41c4193bb7176ad117d08da"
format_stamp: "Formatted at 2026-09-25 15:19:43 on dist-test-slave-8vw7"
server_key: "2138d71062653fa1c35d2c25e563aa22"
server_key_iv: "2c6545450093c06f46f715035e4bed08"
server_key_version: "encryptionkey@0"
I20260925 15:19:43.205132 17050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "f0147c8ca41c4193bb7176ad117d08da"
format_stamp: "Formatted at 2026-09-25 15:19:43 on dist-test-slave-8vw7"
server_key: "2138d71062653fa1c35d2c25e563aa22"
server_key_iv: "2c6545450093c06f46f715035e4bed08"
server_key_version: "encryptionkey@0"
I20260925 15:19:43.209843 17050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:19:43.213285 17065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:43.214385 17050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:19:43.214498 17050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "f0147c8ca41c4193bb7176ad117d08da"
format_stamp: "Formatted at 2026-09-25 15:19:43 on dist-test-slave-8vw7"
server_key: "2138d71062653fa1c35d2c25e563aa22"
server_key_iv: "2c6545450093c06f46f715035e4bed08"
server_key_version: "encryptionkey@0"
I20260925 15:19:43.214654 17050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:43.256561 17050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:43.260563 17050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:43.260778 17050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:43.263252 17050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:43.263355 17050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:43.263494 17050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:43.263562 17050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:19:43 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349583, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:43.287096 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:43.264596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.8.226.67:52923)
0925 15:19:43.265443 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:43.265482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:43.267116 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:43.267850 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:43.267857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:43.268548 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:43.269468 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:43.269495 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.270835 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.270840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:43.273071 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:43.273082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.273898 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.273905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:43.274198 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:43.274924 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:43.274963 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:43.278346 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:43.282868 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:43.282877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:43.282891 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:43.283272 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:43.283707 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:43.283710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:43.283712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:43.283894 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:43.284579 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:43.284584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:43.284856 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 15:19:43.285121 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 15:19:43.285657 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":371,"threads_started":1}
I20260925 15:19:43.288101 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:43.264335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:52923 (local address 127.8.226.126:33777)
0925 15:19:43.264503 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:43.264507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:43.265583 (+ 1076us) server_negotiation.cc:407] Connection header received
0925 15:19:43.267425 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:43.267430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:43.267524 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:43.267627 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:43.269629 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.270672 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.273205 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.273675 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.273804 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:43.278644 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:43.278651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:43.278656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:43.278708 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:43.282730 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:43.283399 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:43.283407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:43.283414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:43.283595 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:43.284001 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:43.284008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:43.284013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:43.284460 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:43.284626 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:43.287703 (+ 3077us) server_negotiation.cc:299] Negotiation successful
0925 15:19:43.287875 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:19:43.307192 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:43.293286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:51327 (local address 127.8.226.126:33777)
0925 15:19:43.293476 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:43.293483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:43.294175 (+ 692us) server_negotiation.cc:407] Connection header received
0925 15:19:43.294274 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:43.294281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:43.294372 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:43.294483 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:43.295311 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.296627 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.298423 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.299011 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.299131 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:43.301713 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:43.301718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:43.301721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:43.301749 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:43.304451 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:43.305341 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:43.305347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:43.305351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:43.305522 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:43.305909 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:43.305914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:43.305917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:43.306143 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:43.306298 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:43.306664 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:19:43.306857 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:19:43.307268 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:43.293411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.8.226.67:51327)
0925 15:19:43.294066 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:43.294083 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:43.294186 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:43.294533 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:43.294538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:43.294936 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:43.295188 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:43.295195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.296779 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.296783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:43.298266 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:43.298288 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.299180 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.299187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:43.299332 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:43.300081 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:43.300110 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:43.301539 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:43.304589 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:43.304596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:43.304603 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:43.305218 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:43.305662 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:43.305665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:43.305667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:43.305786 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:43.306253 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:43.306258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:43.306423 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:19:43.306549 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:19:43.306852 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":373,"threads_started":1}
I20260925 15:19:43.316972 17050 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40725
I20260925 15:19:43.317027 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40725 every 8 connection(s)
I20260925 15:19:43.318173 17050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:43.319995 17185 heartbeater.cc:347] Connected to a master server at 127.8.226.126:33777
I20260925 15:19:43.320248 17185 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:43.320843 17185 heartbeater.cc:511] Master 127.8.226.126:33777 requested a full tablet report, sending...
I20260925 15:19:43.322073 16700 ts_manager.cc:194] Registered new tserver with Master: f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:43.322829 16700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:52923
I20260925 15:19:43.325215 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17050
I20260925 15:19:43.325384 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:43.325812 9097 external_mini_cluster.cc:1428] Setting key 0b12fd3a484f158be977060fcf498008
I20260925 15:19:43.328240 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:43.348686 16907 heartbeater.cc:503] Master 127.8.226.126:33777 was elected leader, sending a full tablet report...
Sep 25 15:19:43 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:19:43 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:43 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:43.816926 17046 heartbeater.cc:503] Master 127.8.226.126:33777 was elected leader, sending a full tablet report...
I20260925 15:19:44.324755 17185 heartbeater.cc:503] Master 127.8.226.126:33777 was elected leader, sending a full tablet report...
W20260925 15:19:44.399964 17193 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.031s user 0.004s sys 0.011s
W20260925 15:19:44.401252 16795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:43.363754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.8.226.126:33777)
0925 15:19:43.363963 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:43.363968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:43.364281 (+ 313us) server_negotiation.cc:407] Connection header received
0925 15:19:43.364469 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:43.364473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:43.364533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:43.364613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:43.365895 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.366574 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.367679 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:43.368073 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:43.368157 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.400919 (+1032762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58742: BlockingRecv error: recv got EOF from 127.0.0.1:58742 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:19:44 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:44.423005 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.405035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.8.226.126:33777)
0925 15:19:44.405216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.405224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.405543 (+ 319us) server_negotiation.cc:407] Connection header received
0925 15:19:44.405754 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.405759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.405832 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.405934 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:44.407053 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.407713 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.408845 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.409311 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.409436 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.418219 (+ 8783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:44.418227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:44.418232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:44.418269 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:44.420857 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.421479 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.421483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.421485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.421594 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.421917 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.421920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.421921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.422186 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:44.422385 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.422595 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.422777 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:19:44.426877 16700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:44.433252 16700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:44.456519 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.444974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.8.226.65:37857)
0925 15:19:44.448336 (+ 3362us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.448346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.448371 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:44.448482 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.448491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.449218 (+ 727us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.450031 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.450339 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.451876 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.454591 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.455468 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.455624 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.455819 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.456070 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.456229 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3161,"thread_start_us":2977,"threads_started":1}
I20260925 15:19:44.456640 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.444476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37857 (local address 127.0.0.1:34454)
0925 15:19:44.445292 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.445383 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.445639 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.449444 (+ 3805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.449450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.449477 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.450133 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.450150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.452020 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.452024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.454449 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.454459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.455648 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.455654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.455834 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.455892 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.456027 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.456213 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":149,"spinlock_wait_cycles":8832,"thread_start_us":688,"threads_started":1}
I20260925 15:19:44.457484 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.444880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.8.226.67:40725)
0925 15:19:44.448931 (+ 4051us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.448939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.448991 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:19:44.449074 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.449080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.449246 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.449410 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.450423 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.451385 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.454363 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.455671 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.455793 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.455945 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.457043 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.457201 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3910,"thread_start_us":3788,"threads_started":1}
I20260925 15:19:44.457556 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.446144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:35325 (local address 127.0.0.1:47424)
0925 15:19:44.448647 (+ 2503us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.448673 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.448854 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.449570 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.449574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.449619 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.450359 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.450370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.452102 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.452107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.454040 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.454056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.456116 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.456123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.456987 (+ 864us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.457018 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.457100 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.457208 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2372,"mutex_wait_us":72,"spinlock_wait_cycles":1408,"thread_start_us":1150,"threads_started":1}
I20260925 15:19:44.457790 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.444617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40725 (local address 127.0.0.1:51152)
0925 15:19:44.447588 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.447616 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.447746 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.449627 (+ 1881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.449632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.449655 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.450256 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.450291 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.451530 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.451536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.454214 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.454227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.456131 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.456136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.456289 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.456319 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.456409 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.456535 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"mutex_wait_us":62,"thread_start_us":3715,"threads_started":1}
I20260925 15:19:44.458209 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.446403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.8.226.66:35325)
0925 15:19:44.446996 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.447008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.448765 (+ 1757us) server_negotiation.cc:407] Connection header received
0925 15:19:44.449084 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.449091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.449315 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.449479 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.450500 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.451939 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.454202 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.455621 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.455765 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.455948 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.457743 (+ 1795us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.457929 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":250,"threads_started":1}
I20260925 15:19:44.460235 16841 tablet_service.cc:1467] Processing CreateTablet for tablet ca97165cc2894bd294a049ae13992857 (DEFAULT_TABLE table=test-table [id=77ac119876c14ece9b2523bc8bd11c01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:44.460510 17115 tablet_service.cc:1467] Processing CreateTablet for tablet ca97165cc2894bd294a049ae13992857 (DEFAULT_TABLE table=test-table [id=77ac119876c14ece9b2523bc8bd11c01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:44.461585 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca97165cc2894bd294a049ae13992857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.461834 17115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca97165cc2894bd294a049ae13992857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.463733 16976 tablet_service.cc:1467] Processing CreateTablet for tablet ca97165cc2894bd294a049ae13992857 (DEFAULT_TABLE table=test-table [id=77ac119876c14ece9b2523bc8bd11c01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:44.464893 16976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca97165cc2894bd294a049ae13992857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.472573 17210 tablet_bootstrap.cc:492] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Bootstrap starting.
I20260925 15:19:44.475673 17210 tablet_bootstrap.cc:654] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.475800 17211 tablet_bootstrap.cc:492] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Bootstrap starting.
I20260925 15:19:44.477378 17210 log.cc:824] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:44.478444 17212 tablet_bootstrap.cc:492] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Bootstrap starting.
I20260925 15:19:44.479787 17211 tablet_bootstrap.cc:654] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.480651 17211 log.cc:824] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:44.481729 17210 tablet_bootstrap.cc:492] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: No bootstrap required, opened a new log
I20260925 15:19:44.481743 17212 tablet_bootstrap.cc:654] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.482038 17210 ts_tablet_manager.cc:1397] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 15:19:44.482852 17212 log.cc:824] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:44.482977 17211 tablet_bootstrap.cc:492] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: No bootstrap required, opened a new log
I20260925 15:19:44.483211 17211 ts_tablet_manager.cc:1397] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:19:44.485554 17212 tablet_bootstrap.cc:492] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: No bootstrap required, opened a new log
I20260925 15:19:44.485890 17212 ts_tablet_manager.cc:1397] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260925 15:19:44.491010 17211 raft_consensus.cc:348] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.491355 17211 raft_consensus.cc:374] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.491451 17211 raft_consensus.cc:729] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0147c8ca41c4193bb7176ad117d08da, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.491871 17210 raft_consensus.cc:348] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.492054 17211 consensus_queue.cc:260] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.492280 17210 raft_consensus.cc:374] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.492415 17210 raft_consensus.cc:729] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.493419 17210 consensus_queue.cc:260] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.493753 17211 ts_tablet_manager.cc:1428] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260925 15:19:44.497488 17212 raft_consensus.cc:348] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.497999 17212 raft_consensus.cc:374] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.498153 17212 raft_consensus.cc:729] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.498987 17212 consensus_queue.cc:260] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.501104 17212 ts_tablet_manager.cc:1428] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260925 15:19:44.502345 17210 ts_tablet_manager.cc:1428] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
W20260925 15:19:44.561641 17047 tablet.cc:2369] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:19:44.570688 17186 tablet.cc:2369] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:19:44.593020 16908 tablet.cc:2369] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:44.627391 17218 raft_consensus.cc:482] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:44.627713 17218 raft_consensus.cc:504] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.629451 17218 leader_election.cc:288] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:44.636621 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.629875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:52993 (local address 127.8.226.65:37857)
0925 15:19:44.630341 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.630347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.630978 (+ 631us) server_negotiation.cc:407] Connection header received
0925 15:19:44.631168 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.631172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.631280 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.631381 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.631974 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.632698 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.634691 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.635637 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.635761 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.635813 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.636226 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.636367 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260925 15:19:44.636639 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.630094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:48343 (local address 127.8.226.67:40725)
0925 15:19:44.630428 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.630436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.630682 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:19:44.630933 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.630941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.631034 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.631128 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.631763 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.632536 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.634773 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.635775 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.635938 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.636009 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.636290 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.636414 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260925 15:19:44.636771 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.629830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37857 (local address 127.8.226.66:52993)
0925 15:19:44.630871 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.630888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.631011 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.631402 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.631405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.631423 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.631809 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.631824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.632843 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.632847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.634480 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.634514 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.635860 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.635869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.636039 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.636077 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.636154 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.636311 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":37,"thread_start_us":733,"threads_started":1}
I20260925 15:19:44.636835 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.629734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40725 (local address 127.8.226.66:48343)
0925 15:19:44.630546 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.630573 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.630764 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.631152 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.631156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.631182 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.631597 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.631614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.632657 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.632663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.634509 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.634523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.635988 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.635992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.636094 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.636120 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.636250 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.636367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":42,"thread_start_us":352,"threads_started":1}
I20260925 15:19:44.637890 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca97165cc2894bd294a049ae13992857" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da" is_pre_election: true
I20260925 15:19:44.637890 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca97165cc2894bd294a049ae13992857" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38da62da3ec4492bb1ada9e416387d3" is_pre_election: true
I20260925 15:19:44.638350 16862 raft_consensus.cc:2454] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 0.
I20260925 15:19:44.638350 17135 raft_consensus.cc:2454] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 0.
I20260925 15:19:44.638967 16928 leader_election.cc:302] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57ce821999cb4f4aa8bf48d6024d6ad7, d38da62da3ec4492bb1ada9e416387d3, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:44.639345 17218 raft_consensus.cc:2781] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:44.639590 17218 raft_consensus.cc:482] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:44.639716 17218 raft_consensus.cc:3037] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.642095 17218 raft_consensus.cc:504] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.642665 17218 leader_election.cc:288] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 election: Requested vote from peers d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:44.643067 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca97165cc2894bd294a049ae13992857" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38da62da3ec4492bb1ada9e416387d3"
I20260925 15:19:44.643088 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca97165cc2894bd294a049ae13992857" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da"
I20260925 15:19:44.643265 16862 raft_consensus.cc:3037] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.643285 17135 raft_consensus.cc:3037] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.645501 16862 raft_consensus.cc:2454] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 1.
I20260925 15:19:44.645514 17135 raft_consensus.cc:2454] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 1.
I20260925 15:19:44.645874 16928 leader_election.cc:302] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ce821999cb4f4aa8bf48d6024d6ad7, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:44.646209 17218 raft_consensus.cc:2781] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:44.646603 17218 raft_consensus.cc:686] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Running, Role: LEADER
I20260925 15:19:44.647094 17218 consensus_queue.cc:237] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } }
I20260925 15:19:44.651487 16699 catalog_manager.cc:5495] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } health_report { overall_health: HEALTHY } } }
I20260925 15:19:44.692441 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.687237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.8.226.66:35325)
0925 15:19:44.687430 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.687438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.687508 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:19:44.687769 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.687773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.687856 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.688006 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:44.688360 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.689088 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.690422 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.690819 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.690976 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.691382 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:44.691887 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:44.692038 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.692108 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.692181 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:19:44.700282 16862 raft_consensus.cc:1261] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Refusing update from remote peer 57ce821999cb4f4aa8bf48d6024d6ad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:44.700325 17135 raft_consensus.cc:1261] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Refusing update from remote peer 57ce821999cb4f4aa8bf48d6024d6ad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:44.701242 17218 consensus_queue.cc:1035] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:44.701545 17221 consensus_queue.cc:1035] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:44.713003 17225 mvcc.cc:205] Tried to move back new op lower bound from 7333271898922504192 to 7333271898717974528. Current Snapshot: MvccSnapshot[applied={T|T < 7333271898922504192}]
I20260925 15:19:44.713481 17224 mvcc.cc:205] Tried to move back new op lower bound from 7333271898922504192 to 7333271898717974528. Current Snapshot: MvccSnapshot[applied={T|T < 7333271898922504192}]
I20260925 15:19:44.728075 16699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:44.728360 16699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:44.731926 16699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Sending DeleteTablet for 3 replicas of tablet ca97165cc2894bd294a049ae13992857
I20260925 15:19:44.733036 17115 tablet_service.cc:1514] Processing DeleteTablet for tablet ca97165cc2894bd294a049ae13992857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:44 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:51152
I20260925 15:19:44.733201 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet ca97165cc2894bd294a049ae13992857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:44 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34454
I20260925 15:19:44.734171 16976 tablet_service.cc:1514] Processing DeleteTablet for tablet ca97165cc2894bd294a049ae13992857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:44 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47424
I20260925 15:19:44.734176 16700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:44.734803 16700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:44.742211 16840 tablet_service.cc:1467] Processing CreateTablet for tablet 8934e2091b294ab79bddd736876d7d66 (DEFAULT_TABLE table=test-table [id=6b239c2f7b4a4d69bc9a201c5c433665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:44.742211 16975 tablet_service.cc:1467] Processing CreateTablet for tablet 8934e2091b294ab79bddd736876d7d66 (DEFAULT_TABLE table=test-table [id=6b239c2f7b4a4d69bc9a201c5c433665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:44.742774 16840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8934e2091b294ab79bddd736876d7d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.743453 17114 tablet_service.cc:1467] Processing CreateTablet for tablet 8934e2091b294ab79bddd736876d7d66 (DEFAULT_TABLE table=test-table [id=6b239c2f7b4a4d69bc9a201c5c433665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:44.743954 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8934e2091b294ab79bddd736876d7d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.745405 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8934e2091b294ab79bddd736876d7d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.751753 17233 tablet_replica.cc:330] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: stopping tablet replica
I20260925 15:19:44.752413 17233 raft_consensus.cc:2229] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:44.753021 17233 raft_consensus.cc:2258] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:44.753240 17210 tablet_bootstrap.cc:492] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Bootstrap starting.
I20260925 15:19:44.753522 17211 tablet_bootstrap.cc:492] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Bootstrap starting.
I20260925 15:19:44.754959 17210 tablet_bootstrap.cc:654] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.755175 17211 tablet_bootstrap.cc:654] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.755782 17233 ts_tablet_manager.cc:1905] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:44.757861 17210 tablet_bootstrap.cc:492] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: No bootstrap required, opened a new log
I20260925 15:19:44.758001 17210 ts_tablet_manager.cc:1397] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:19:44.759790 17210 raft_consensus.cc:348] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.760027 17210 raft_consensus.cc:374] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.760123 17210 raft_consensus.cc:729] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.760453 17210 consensus_queue.cc:260] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.761134 17210 ts_tablet_manager.cc:1428] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:44.762259 17212 tablet_bootstrap.cc:492] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Bootstrap starting.
I20260925 15:19:44.763728 17212 tablet_bootstrap.cc:654] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.764513 17235 tablet_replica.cc:330] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: stopping tablet replica
I20260925 15:19:44.765033 17235 raft_consensus.cc:2229] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:44.765708 17235 raft_consensus.cc:2258] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:44.768160 17235 ts_tablet_manager.cc:1905] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:44.773236 17234 tablet_replica.cc:330] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: stopping tablet replica
I20260925 15:19:44.773620 17234 raft_consensus.cc:2229] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:44.774039 17234 raft_consensus.cc:2258] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:44.775928 17234 ts_tablet_manager.cc:1905] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:44.778247 17211 tablet_bootstrap.cc:492] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: No bootstrap required, opened a new log
I20260925 15:19:44.778502 17211 ts_tablet_manager.cc:1397] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.001s
I20260925 15:19:44.779585 17211 raft_consensus.cc:348] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.779958 17211 raft_consensus.cc:374] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.780115 17211 raft_consensus.cc:729] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0147c8ca41c4193bb7176ad117d08da, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.781883 17211 consensus_queue.cc:260] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.782626 17234 ts_tablet_manager.cc:1918] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:44.782837 17234 log.cc:1196] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/ca97165cc2894bd294a049ae13992857
I20260925 15:19:44.783375 17234 ts_tablet_manager.cc:1939] T ca97165cc2894bd294a049ae13992857 P d38da62da3ec4492bb1ada9e416387d3: Deleting consensus metadata
I20260925 15:19:44.783914 17235 ts_tablet_manager.cc:1918] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:44.785185 16686 catalog_manager.cc:4841] TS d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857): tablet ca97165cc2894bd294a049ae13992857 (table test-table [id=77ac119876c14ece9b2523bc8bd11c01]) successfully deleted
I20260925 15:19:44.785459 17211 ts_tablet_manager.cc:1428] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260925 15:19:44.789145 17235 log.cc:1196] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/ca97165cc2894bd294a049ae13992857
I20260925 15:19:44.787074 17212 tablet_bootstrap.cc:492] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: No bootstrap required, opened a new log
I20260925 15:19:44.789678 17235 ts_tablet_manager.cc:1939] T ca97165cc2894bd294a049ae13992857 P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting consensus metadata
I20260925 15:19:44.789834 17212 ts_tablet_manager.cc:1397] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.012s
I20260925 15:19:44.790036 17233 ts_tablet_manager.cc:1918] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:44.790284 17233 log.cc:1196] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/ca97165cc2894bd294a049ae13992857
I20260925 15:19:44.790693 17212 raft_consensus.cc:348] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.790980 17212 raft_consensus.cc:374] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.791023 17233 ts_tablet_manager.cc:1939] T ca97165cc2894bd294a049ae13992857 P f0147c8ca41c4193bb7176ad117d08da: Deleting consensus metadata
I20260925 15:19:44.791121 17212 raft_consensus.cc:729] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.791433 17212 consensus_queue.cc:260] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.792057 17212 ts_tablet_manager.cc:1428] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:19:44.792471 16684 catalog_manager.cc:4841] TS 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325): tablet ca97165cc2894bd294a049ae13992857 (table test-table [id=77ac119876c14ece9b2523bc8bd11c01]) successfully deleted
I20260925 15:19:44.792869 16685 catalog_manager.cc:4841] TS f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725): tablet ca97165cc2894bd294a049ae13992857 (table test-table [id=77ac119876c14ece9b2523bc8bd11c01]) successfully deleted
I20260925 15:19:44.846783 17218 raft_consensus.cc:482] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:44.846997 17218 raft_consensus.cc:504] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.847738 17218 leader_election.cc:288] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:44.848272 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8934e2091b294ab79bddd736876d7d66" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da" is_pre_election: true
I20260925 15:19:44.848256 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8934e2091b294ab79bddd736876d7d66" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38da62da3ec4492bb1ada9e416387d3" is_pre_election: true
I20260925 15:19:44.848526 17135 raft_consensus.cc:2454] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 0.
I20260925 15:19:44.848526 16862 raft_consensus.cc:2454] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 0.
I20260925 15:19:44.848994 16928 leader_election.cc:302] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ce821999cb4f4aa8bf48d6024d6ad7, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:44.849279 17218 raft_consensus.cc:2781] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:44.849419 17218 raft_consensus.cc:482] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:44.849512 17218 raft_consensus.cc:3037] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.852656 17218 raft_consensus.cc:504] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.853312 17218 leader_election.cc:288] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 election: Requested vote from peers d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:44.853688 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8934e2091b294ab79bddd736876d7d66" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38da62da3ec4492bb1ada9e416387d3"
I20260925 15:19:44.853718 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8934e2091b294ab79bddd736876d7d66" candidate_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da"
I20260925 15:19:44.853884 16862 raft_consensus.cc:3037] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.853899 17135 raft_consensus.cc:3037] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.855798 17135 raft_consensus.cc:2454] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 1.
I20260925 15:19:44.856176 16928 leader_election.cc:302] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ce821999cb4f4aa8bf48d6024d6ad7, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:44.856467 17218 raft_consensus.cc:2781] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:44.856635 17218 raft_consensus.cc:686] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Running, Role: LEADER
I20260925 15:19:44.856761 16862 raft_consensus.cc:2454] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ce821999cb4f4aa8bf48d6024d6ad7 in term 1.
I20260925 15:19:44.856925 17218 consensus_queue.cc:237] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:44.859566 16700 catalog_manager.cc:5495] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:19:44 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349581, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.126@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:44.910297 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.895340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:33777 (local address 127.0.0.1:58750)
0925 15:19:44.895984 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.896005 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.896133 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.896495 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.896498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.897215 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:44.897466 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.897474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.898900 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.898903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.900055 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.900066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.901065 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.901069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.901175 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.901875 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:44.901892 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:44.904320 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:44.907888 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:44.907898 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:44.907904 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:44.908328 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:44.908673 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:44.908676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:44.908678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:44.908806 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:44.909404 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:44.909410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:44.909567 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.909712 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.909886 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":365,"threads_started":1}
I20260925 15:19:44.910516 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.895329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.8.226.126:33777)
0925 15:19:44.895563 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.895570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.896097 (+ 527us) server_negotiation.cc:407] Connection header received
0925 15:19:44.896265 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.896270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.896349 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.896455 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:44.897603 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.898779 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.900187 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.900820 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.900990 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.904486 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:44.904494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:44.904501 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:44.904537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:44.907762 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.908433 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.908439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.908442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.908571 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.908902 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.908907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.908911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.909282 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:44.909438 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.909834 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.910044 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 15:19:44.915400 16700 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:58750:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:19:44.916452 16700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:44.923652 16975 tablet_service.cc:1467] Processing CreateTablet for tablet 3d44daace9b1449bbcae0f648f378980 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e1b77a626c445a796e3277d885f4e1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:19:44.923972 16841 tablet_service.cc:1467] Processing CreateTablet for tablet 3d44daace9b1449bbcae0f648f378980 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e1b77a626c445a796e3277d885f4e1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:19:44.924153 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d44daace9b1449bbcae0f648f378980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.924479 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d44daace9b1449bbcae0f648f378980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.929579 17114 tablet_service.cc:1467] Processing CreateTablet for tablet 3d44daace9b1449bbcae0f648f378980 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e1b77a626c445a796e3277d885f4e1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:19:44.930089 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d44daace9b1449bbcae0f648f378980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.930831 17212 tablet_bootstrap.cc:492] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: Bootstrap starting.
I20260925 15:19:44.931468 17210 tablet_bootstrap.cc:492] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: Bootstrap starting.
I20260925 15:19:44.932552 17212 tablet_bootstrap.cc:654] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.933033 17210 tablet_bootstrap.cc:654] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.937752 17211 tablet_bootstrap.cc:492] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: Bootstrap starting.
I20260925 15:19:44.939741 17211 tablet_bootstrap.cc:654] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.943588 17210 tablet_bootstrap.cc:492] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: No bootstrap required, opened a new log
I20260925 15:19:44.943802 17210 ts_tablet_manager.cc:1397] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.008s
I20260925 15:19:44.944545 17211 tablet_bootstrap.cc:492] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: No bootstrap required, opened a new log
I20260925 15:19:44.944535 17210 raft_consensus.cc:348] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.944713 17211 ts_tablet_manager.cc:1397] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260925 15:19:44.944835 17210 raft_consensus.cc:374] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.944904 17210 raft_consensus.cc:729] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.945215 17210 consensus_queue.cc:260] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.945478 17211 raft_consensus.cc:348] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.945693 17211 raft_consensus.cc:374] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.945763 17211 raft_consensus.cc:729] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0147c8ca41c4193bb7176ad117d08da, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.945638 17217 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:44.945871 17212 tablet_bootstrap.cc:492] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: No bootstrap required, opened a new log
I20260925 15:19:44.945955 17217 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:44.946022 17212 ts_tablet_manager.cc:1397] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260925 15:19:44.945969 17211 consensus_queue.cc:260] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.946503 17211 ts_tablet_manager.cc:1428] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:19:44.946759 17212 raft_consensus.cc:348] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.947021 17212 raft_consensus.cc:374] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.947131 17212 raft_consensus.cc:729] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.947373 17212 consensus_queue.cc:260] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.947716 17216 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:44.948096 17216 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:44.947938 17218 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:44.948220 17218 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:44.948922 17212 ts_tablet_manager.cc:1428] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 15:19:44.949767 17210 ts_tablet_manager.cc:1428] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 15:19:44.981189 17217 raft_consensus.cc:482] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:44.981411 17217 raft_consensus.cc:504] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.983094 17217 leader_election.cc:288] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:44.991662 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.983595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:42595 (local address 127.8.226.66:35325)
0925 15:19:44.983810 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.983818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.984341 (+ 523us) server_negotiation.cc:407] Connection header received
0925 15:19:44.984536 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.984541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.984614 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.984708 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.985391 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.986505 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.989192 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.990452 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.990549 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.990603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.991195 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.991337 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:19:44.992024 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.983373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40725 (local address 127.8.226.65:42237)
0925 15:19:44.984426 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.984440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.984634 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.985194 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.985200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.985225 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.985605 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.985616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.987090 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.987095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.989274 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.989288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.991151 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.991158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.991327 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.991357 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.991447 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.991574 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":150,"thread_start_us":597,"threads_started":1}
I20260925 15:19:44.992151 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.983533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:42237 (local address 127.8.226.67:40725)
0925 15:19:44.983703 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.983709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.984542 (+ 833us) server_negotiation.cc:407] Connection header received
0925 15:19:44.984807 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.984813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.984890 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.984938 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:44.985804 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.986938 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.989485 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.990930 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.991107 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.991182 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.991492 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.991627 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:19:44.992395 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.983373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:35325 (local address 127.8.226.65:42595)
0925 15:19:44.984220 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:44.984240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:44.984387 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:44.984747 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:44.984752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:44.984778 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:44.985159 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.985173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.986640 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.986647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.988940 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:44.988973 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.990808 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.990812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:44.990982 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.991026 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:19:44.991916 (+ 890us) client_negotiation.cc:240] Negotiation successful
0925 15:19:44.992070 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":53,"thread_start_us":511,"threads_started":1}
I20260925 15:19:44.993218 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44daace9b1449bbcae0f648f378980" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da" is_pre_election: true
I20260925 15:19:44.993537 17135 raft_consensus.cc:2454] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 0.
I20260925 15:19:44.993552 16994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44daace9b1449bbcae0f648f378980" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" is_pre_election: true
I20260925 15:19:44.994014 16994 raft_consensus.cc:2454] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 0.
I20260925 15:19:44.994026 16788 leader_election.cc:302] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d38da62da3ec4492bb1ada9e416387d3, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:44.994467 17217 raft_consensus.cc:2781] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:44.994593 17217 raft_consensus.cc:482] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:44.994685 17217 raft_consensus.cc:3037] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.997920 17217 raft_consensus.cc:504] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:44.998605 17217 leader_election.cc:288] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 election: Requested vote from peers 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:44.999081 17135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44daace9b1449bbcae0f648f378980" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da"
I20260925 15:19:44.999071 16994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44daace9b1449bbcae0f648f378980" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7"
I20260925 15:19:44.999274 16994 raft_consensus.cc:3037] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.999315 17135 raft_consensus.cc:3037] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.001323 17135 raft_consensus.cc:2454] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 1.
I20260925 15:19:45.001718 16788 leader_election.cc:302] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d38da62da3ec4492bb1ada9e416387d3, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:45.001991 17217 raft_consensus.cc:2781] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:45.002529 16994 raft_consensus.cc:2454] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 1.
I20260925 15:19:45.010671 17217 raft_consensus.cc:686] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [term 1 LEADER]: Becoming Leader. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Running, Role: LEADER
I20260925 15:19:45.011273 17217 consensus_queue.cc:237] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } }
I20260925 15:19:45.013281 17251 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: TxnStatusTablet state changed. Reason: New leader d38da62da3ec4492bb1ada9e416387d3. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.013731 17251 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:19:45.014559 17253 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:19:45.014777 17253 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:19:45.015172 16697 catalog_manager.cc:5495] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65). New cstate: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } health_report { overall_health: HEALTHY } } }
I20260925 15:19:45.039669 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.032405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37857 (local address 127.0.0.1:34466)
0925 15:19:45.032721 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.032743 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.032879 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.033548 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.033553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.033565 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:45.033737 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.033748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.035114 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.035120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.036681 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.036696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.037798 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.037805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.037993 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.038060 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.039117 (+ 1057us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.039135 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.039206 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.039366 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 15:19:45.040409 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.032585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.8.226.65:37857)
0925 15:19:45.033143 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.033149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.033167 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:19:45.033269 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.033274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.033361 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.033514 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.033872 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.034980 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.036850 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.037521 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.037672 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.038294 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.038929 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.039066 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.040010 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.040090 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442}
I20260925 15:19:45.046149 17135 raft_consensus.cc:1261] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Refusing update from remote peer d38da62da3ec4492bb1ada9e416387d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.046479 16994 raft_consensus.cc:1261] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Refusing update from remote peer d38da62da3ec4492bb1ada9e416387d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.047107 17217 consensus_queue.cc:1035] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.047488 17251 consensus_queue.cc:1035] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.060036 17216 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: TxnStatusTablet state changed. Reason: New leader d38da62da3ec4492bb1ada9e416387d3. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.060379 17216 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:45.062286 17218 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: TxnStatusTablet state changed. Reason: New leader d38da62da3ec4492bb1ada9e416387d3. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.062623 17218 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:45.065205 17216 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.065423 17216 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P f0147c8ca41c4193bb7176ad117d08da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:45.074352 17217 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.075033 17217 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:19:45.084367 17217 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.084635 17217 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P d38da62da3ec4492bb1ada9e416387d3: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:19:45.089380 17218 tablet_replica.cc:439] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } } peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } }
I20260925 15:19:45.089677 17218 tablet_replica.cc:442] T 3d44daace9b1449bbcae0f648f378980 P 57ce821999cb4f4aa8bf48d6024d6ad7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:19:45.102563 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.094926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:48619 (local address 127.8.226.65:37857)
0925 15:19:45.095086 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.095092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.095687 (+ 595us) server_negotiation.cc:407] Connection header received
0925 15:19:45.095897 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.095903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.095986 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.096095 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.096765 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.097957 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.100172 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.100812 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.100916 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.101651 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.101790 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.101903 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.102046 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.102133 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:19:45.102568 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.094537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37857 (local address 127.8.226.66:48619)
0925 15:19:45.095559 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.095582 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.095731 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.096135 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.096140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.096153 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:45.096556 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.096568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.098109 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.098114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.100018 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.100032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.101129 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.101135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.101368 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.101432 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.101907 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.101918 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.102004 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.102152 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":718,"threads_started":1}
I20260925 15:19:45.112870 17134 raft_consensus.cc:1261] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Refusing update from remote peer 57ce821999cb4f4aa8bf48d6024d6ad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.113307 16862 raft_consensus.cc:1261] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Refusing update from remote peer 57ce821999cb4f4aa8bf48d6024d6ad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.113986 17221 consensus_queue.cc:1035] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.114434 17221 consensus_queue.cc:1035] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.120151 17226 mvcc.cc:205] Tried to move back new op lower bound from 7333271900616212480 to 7333271899576569856. Current Snapshot: MvccSnapshot[applied={T|T < 7333271900616212480}]
I20260925 15:19:45.166051 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.159299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:57479 (local address 127.8.226.66:35325)
0925 15:19:45.159478 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.159486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.160170 (+ 684us) server_negotiation.cc:407] Connection header received
0925 15:19:45.160387 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.160393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.160507 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.160602 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.161208 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.162349 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.164239 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.164664 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.164821 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.165303 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.165403 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.165500 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.165670 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.165743 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:19:45.166136 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.159319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:35325 (local address 127.8.226.65:57479)
0925 15:19:45.160052 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.160082 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.160252 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.160617 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.160622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.160634 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:45.161049 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.161061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.162510 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.162515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.164106 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.164118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.164881 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.164888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.165070 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.165125 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.165500 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.165512 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.165629 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.165771 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":397,"threads_started":1}
I20260925 15:19:45.204512 16699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:45.204734 16699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:45.207569 16699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Sending DeleteTablet for 3 replicas of tablet 8934e2091b294ab79bddd736876d7d66
I20260925 15:19:45.208191 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet 8934e2091b294ab79bddd736876d7d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34454
I20260925 15:19:45.208469 17234 tablet_replica.cc:330] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: stopping tablet replica
I20260925 15:19:45.208675 17234 raft_consensus.cc:2229] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:45.208750 17114 tablet_service.cc:1514] Processing DeleteTablet for tablet 8934e2091b294ab79bddd736876d7d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:51152
I20260925 15:19:45.208735 16975 tablet_service.cc:1514] Processing DeleteTablet for tablet 8934e2091b294ab79bddd736876d7d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47424
I20260925 15:19:45.208856 17234 raft_consensus.cc:2258] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.208988 17233 tablet_replica.cc:330] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: stopping tablet replica
I20260925 15:19:45.209088 17235 tablet_replica.cc:330] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: stopping tablet replica
I20260925 15:19:45.209208 17233 raft_consensus.cc:2229] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:45.209291 17235 raft_consensus.cc:2229] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:45.209394 17233 raft_consensus.cc:2258] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.209580 17235 raft_consensus.cc:2258] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.210882 17233 ts_tablet_manager.cc:1905] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.211931 17234 ts_tablet_manager.cc:1905] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.213893 17235 ts_tablet_manager.cc:1905] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.220178 17234 ts_tablet_manager.cc:1918] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:45.220337 17234 log.cc:1196] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/8934e2091b294ab79bddd736876d7d66
I20260925 15:19:45.220777 17234 ts_tablet_manager.cc:1939] T 8934e2091b294ab79bddd736876d7d66 P d38da62da3ec4492bb1ada9e416387d3: Deleting consensus metadata
I20260925 15:19:45.221864 16686 catalog_manager.cc:4841] TS d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857): tablet 8934e2091b294ab79bddd736876d7d66 (table test-table [id=6b239c2f7b4a4d69bc9a201c5c433665]) successfully deleted
I20260925 15:19:45.225003 17235 ts_tablet_manager.cc:1918] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:45.227648 17233 ts_tablet_manager.cc:1918] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:45.227813 17233 log.cc:1196] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/8934e2091b294ab79bddd736876d7d66
I20260925 15:19:45.228219 17233 ts_tablet_manager.cc:1939] T 8934e2091b294ab79bddd736876d7d66 P f0147c8ca41c4193bb7176ad117d08da: Deleting consensus metadata
I20260925 15:19:45.229092 17235 log.cc:1196] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/8934e2091b294ab79bddd736876d7d66
I20260925 15:19:45.229122 16685 catalog_manager.cc:4841] TS f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725): tablet 8934e2091b294ab79bddd736876d7d66 (table test-table [id=6b239c2f7b4a4d69bc9a201c5c433665]) successfully deleted
I20260925 15:19:45.229519 17235 ts_tablet_manager.cc:1939] T 8934e2091b294ab79bddd736876d7d66 P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting consensus metadata
I20260925 15:19:45.230528 16684 catalog_manager.cc:4841] TS 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325): tablet 8934e2091b294ab79bddd736876d7d66 (table test-table [id=6b239c2f7b4a4d69bc9a201c5c433665]) successfully deleted
Sep 25 15:19:45 dist-test-slave-8vw7 krb5kdc[16655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:19:45 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:45 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:19:45.274356 16795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:45.263217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.8.226.126:33777)
0925 15:19:45.263438 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.263448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.264082 (+ 634us) server_negotiation.cc:407] Connection header received
0925 15:19:45.264312 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.264317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.264384 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.264504 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:45.266010 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.266721 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.267944 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.268374 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.268532 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.274078 (+ 5546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58766: BlockingRecv error: recv got EOF from 127.0.0.1:58766 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 15:19:45 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM
I20260925 15:19:45.294703 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.278024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.8.226.126:33777)
0925 15:19:45.278252 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.278260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.281453 (+ 3193us) server_negotiation.cc:407] Connection header received
0925 15:19:45.281640 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.281646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.281717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.281820 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:45.282979 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.283693 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.284875 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.285350 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.285474 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.289114 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:45.289119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:45.289122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:45.289177 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:45.291987 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:45.292600 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:45.292606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:45.292610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:45.292782 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:45.293194 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:45.293200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:45.293204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:45.293455 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:45.293732 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.294041 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.294309 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:19:45.298003 16699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:45.298632 16699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:45.305590 16841 tablet_service.cc:1467] Processing CreateTablet for tablet 34f848691faa466ea696c97be5e7d79f (DEFAULT_TABLE table=test-table [id=f54eb59e8d034b7da22e40f414736ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:45.306048 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f848691faa466ea696c97be5e7d79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:45.309360 16975 tablet_service.cc:1467] Processing CreateTablet for tablet 34f848691faa466ea696c97be5e7d79f (DEFAULT_TABLE table=test-table [id=f54eb59e8d034b7da22e40f414736ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:45.309791 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f848691faa466ea696c97be5e7d79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:45.311564 17210 tablet_bootstrap.cc:492] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Bootstrap starting.
I20260925 15:19:45.313002 17210 tablet_bootstrap.cc:654] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:45.315317 17210 tablet_bootstrap.cc:492] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: No bootstrap required, opened a new log
I20260925 15:19:45.315472 17210 ts_tablet_manager.cc:1397] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:45.315737 17212 tablet_bootstrap.cc:492] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Bootstrap starting.
I20260925 15:19:45.316279 17210 raft_consensus.cc:348] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.316550 17210 raft_consensus.cc:374] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:45.316641 17210 raft_consensus.cc:729] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Initialized, Role: FOLLOWER
I20260925 15:19:45.317253 17212 tablet_bootstrap.cc:654] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:45.317260 17210 consensus_queue.cc:260] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.317827 17210 ts_tablet_manager.cc:1428] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:45.318420 17114 tablet_service.cc:1467] Processing CreateTablet for tablet 34f848691faa466ea696c97be5e7d79f (DEFAULT_TABLE table=test-table [id=f54eb59e8d034b7da22e40f414736ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:45.318856 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f848691faa466ea696c97be5e7d79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:45.324980 17211 tablet_bootstrap.cc:492] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Bootstrap starting.
I20260925 15:19:45.326682 17211 tablet_bootstrap.cc:654] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:45.334461 17212 tablet_bootstrap.cc:492] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: No bootstrap required, opened a new log
I20260925 15:19:45.334707 17212 ts_tablet_manager.cc:1397] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20260925 15:19:45.335814 17212 raft_consensus.cc:348] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.337424 17212 raft_consensus.cc:374] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:45.337592 17212 raft_consensus.cc:729] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Initialized, Role: FOLLOWER
I20260925 15:19:45.337939 17212 consensus_queue.cc:260] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.338743 17212 ts_tablet_manager.cc:1428] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 15:19:45.346398 17211 tablet_bootstrap.cc:492] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: No bootstrap required, opened a new log
I20260925 15:19:45.346596 17211 ts_tablet_manager.cc:1397] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.010s
I20260925 15:19:45.347379 17211 raft_consensus.cc:348] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.347707 17211 raft_consensus.cc:374] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:45.347831 17211 raft_consensus.cc:729] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0147c8ca41c4193bb7176ad117d08da, State: Initialized, Role: FOLLOWER
I20260925 15:19:45.348083 17211 consensus_queue.cc:260] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.348645 17211 ts_tablet_manager.cc:1428] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:19:45.370385 17251 raft_consensus.cc:482] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:45.370594 17251 raft_consensus.cc:504] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.371256 17251 leader_election.cc:288] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:45.371747 16994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f848691faa466ea696c97be5e7d79f" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" is_pre_election: true
I20260925 15:19:45.371871 17133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f848691faa466ea696c97be5e7d79f" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da" is_pre_election: true
I20260925 15:19:45.371982 16994 raft_consensus.cc:2454] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 0.
I20260925 15:19:45.372114 17133 raft_consensus.cc:2454] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 0.
I20260925 15:19:45.372601 16788 leader_election.cc:302] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d38da62da3ec4492bb1ada9e416387d3, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:45.372939 17251 raft_consensus.cc:2781] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:45.373099 17251 raft_consensus.cc:482] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:45.373226 17251 raft_consensus.cc:3037] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.375967 17251 raft_consensus.cc:504] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.376744 17251 leader_election.cc:288] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 election: Requested vote from peers 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325), f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725)
I20260925 15:19:45.376945 16994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f848691faa466ea696c97be5e7d79f" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7"
I20260925 15:19:45.377182 16994 raft_consensus.cc:3037] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.377316 17133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f848691faa466ea696c97be5e7d79f" candidate_uuid: "d38da62da3ec4492bb1ada9e416387d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0147c8ca41c4193bb7176ad117d08da"
I20260925 15:19:45.377488 17133 raft_consensus.cc:3037] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.380100 16994 raft_consensus.cc:2454] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 1.
I20260925 15:19:45.380339 17133 raft_consensus.cc:2454] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38da62da3ec4492bb1ada9e416387d3 in term 1.
I20260925 15:19:45.380553 16787 leader_election.cc:302] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ce821999cb4f4aa8bf48d6024d6ad7, d38da62da3ec4492bb1ada9e416387d3; no voters:
I20260925 15:19:45.380908 17251 raft_consensus.cc:2781] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:45.381125 17251 raft_consensus.cc:686] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 1 LEADER]: Becoming Leader. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Running, Role: LEADER
I20260925 15:19:45.381385 17251 consensus_queue.cc:237] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.383926 16699 catalog_manager.cc:5495] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65). New cstate: current_term: 1 leader_uuid: "d38da62da3ec4492bb1ada9e416387d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:45.401319 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.396303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.8.226.65:37857)
0925 15:19:45.396499 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.396506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.396664 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:19:45.396870 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.396876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.396942 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.397062 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.397408 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.398266 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.399597 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.400013 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.400166 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.400568 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.400682 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.400780 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.400927 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.401031 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:19:45.404563 16994 raft_consensus.cc:1261] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Refusing update from remote peer d38da62da3ec4492bb1ada9e416387d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.404619 17133 raft_consensus.cc:1261] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Refusing update from remote peer d38da62da3ec4492bb1ada9e416387d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.405185 17259 consensus_queue.cc:1035] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.405475 17251 consensus_queue.cc:1035] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.424140 16698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:45.424437 16698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:45.427742 16698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Sending DeleteTablet for 3 replicas of tablet 34f848691faa466ea696c97be5e7d79f
I20260925 15:19:45.428525 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f848691faa466ea696c97be5e7d79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34454
I20260925 15:19:45.428537 16975 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f848691faa466ea696c97be5e7d79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47424
I20260925 15:19:45.428846 17235 tablet_replica.cc:330] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: stopping tablet replica
I20260925 15:19:45.429132 17234 tablet_replica.cc:330] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: stopping tablet replica
I20260925 15:19:45.429313 17234 raft_consensus.cc:2229] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:45.429837 17234 raft_consensus.cc:2258] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.430151 16975 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f848691faa466ea696c97be5e7d79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47424
I20260925 15:19:45.430217 16698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:19:45.430409 17235 raft_consensus.cc:2229] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:45.430657 17235 raft_consensus.cc:2258] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 15:19:45.430810 16698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:45.431329 17114 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f848691faa466ea696c97be5e7d79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:51152
I20260925 15:19:45.431615 17233 tablet_replica.cc:330] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: stopping tablet replica
I20260925 15:19:45.431790 17233 raft_consensus.cc:2229] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:45.432003 17233 raft_consensus.cc:2258] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.432250 17234 ts_tablet_manager.cc:1905] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.433139 17233 ts_tablet_manager.cc:1905] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.438117 17114 tablet_service.cc:1467] Processing CreateTablet for tablet 64920551545f4b5d9bd62250b3e104eb (DEFAULT_TABLE table=test-table [id=544014898101483591ecab0df47ca40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:45.438225 16841 tablet_service.cc:1467] Processing CreateTablet for tablet 64920551545f4b5d9bd62250b3e104eb (DEFAULT_TABLE table=test-table [id=544014898101483591ecab0df47ca40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:45.438484 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64920551545f4b5d9bd62250b3e104eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:45.438627 16841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64920551545f4b5d9bd62250b3e104eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:45.439471 17234 ts_tablet_manager.cc:1918] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:45.439580 17234 log.cc:1196] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/34f848691faa466ea696c97be5e7d79f
I20260925 15:19:45.440013 17234 ts_tablet_manager.cc:1939] T 34f848691faa466ea696c97be5e7d79f P d38da62da3ec4492bb1ada9e416387d3: Deleting consensus metadata
I20260925 15:19:45.441051 16686 catalog_manager.cc:4841] TS d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857): tablet 34f848691faa466ea696c97be5e7d79f (table test-table [id=f54eb59e8d034b7da22e40f414736ce4]) successfully deleted
I20260925 15:19:45.441958 17235 ts_tablet_manager.cc:1905] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.445679 17233 ts_tablet_manager.cc:1918] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:45.445858 16975 tablet_service.cc:1467] Processing CreateTablet for tablet 64920551545f4b5d9bd62250b3e104eb (DEFAULT_TABLE table=test-table [id=544014898101483591ecab0df47ca40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:45.446305 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64920551545f4b5d9bd62250b3e104eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:45.445962 17211 tablet_bootstrap.cc:492] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Bootstrap starting.
I20260925 15:19:45.448002 17211 tablet_bootstrap.cc:654] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:45.448446 17235 ts_tablet_manager.cc:1918] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:45.448561 17235 log.cc:1196] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/34f848691faa466ea696c97be5e7d79f
I20260925 15:19:45.448997 17235 ts_tablet_manager.cc:1939] T 34f848691faa466ea696c97be5e7d79f P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting consensus metadata
I20260925 15:19:45.450222 16684 catalog_manager.cc:4841] TS 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325): tablet 34f848691faa466ea696c97be5e7d79f (table test-table [id=f54eb59e8d034b7da22e40f414736ce4]) successfully deleted
I20260925 15:19:45.451020 17233 log.cc:1196] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/34f848691faa466ea696c97be5e7d79f
W20260925 15:19:45.451545 16684 catalog_manager.cc:4808] TS 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325): delete failed for tablet 34f848691faa466ea696c97be5e7d79f because the tablet was not found. No further retry: Not found: Tablet not found: 34f848691faa466ea696c97be5e7d79f
I20260925 15:19:45.451617 17233 ts_tablet_manager.cc:1939] T 34f848691faa466ea696c97be5e7d79f P f0147c8ca41c4193bb7176ad117d08da: Deleting consensus metadata
I20260925 15:19:45.453064 16685 catalog_manager.cc:4841] TS f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725): tablet 34f848691faa466ea696c97be5e7d79f (table test-table [id=f54eb59e8d034b7da22e40f414736ce4]) successfully deleted
I20260925 15:19:45.453619 17212 tablet_bootstrap.cc:492] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Bootstrap starting.
I20260925 15:19:45.455180 17212 tablet_bootstrap.cc:654] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:45.456642 17210 tablet_bootstrap.cc:492] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Bootstrap starting.
I20260925 15:19:45.458319 17211 tablet_bootstrap.cc:492] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: No bootstrap required, opened a new log
I20260925 15:19:45.458360 17210 tablet_bootstrap.cc:654] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:45.458510 17211 ts_tablet_manager.cc:1397] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260925 15:19:45.459146 17211 raft_consensus.cc:348] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.459403 17211 raft_consensus.cc:374] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:45.459486 17211 raft_consensus.cc:729] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0147c8ca41c4193bb7176ad117d08da, State: Initialized, Role: FOLLOWER
I20260925 15:19:45.459853 17212 tablet_bootstrap.cc:492] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: No bootstrap required, opened a new log
I20260925 15:19:45.459770 17211 consensus_queue.cc:260] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.459997 17212 ts_tablet_manager.cc:1397] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260925 15:19:45.460752 17212 raft_consensus.cc:348] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.460892 17211 ts_tablet_manager.cc:1428] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:19:45.461015 17212 raft_consensus.cc:374] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:45.461097 17212 raft_consensus.cc:729] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ce821999cb4f4aa8bf48d6024d6ad7, State: Initialized, Role: FOLLOWER
I20260925 15:19:45.461359 17212 consensus_queue.cc:260] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.461966 17212 ts_tablet_manager.cc:1428] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:45.470284 17210 tablet_bootstrap.cc:492] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: No bootstrap required, opened a new log
I20260925 15:19:45.470480 17210 ts_tablet_manager.cc:1397] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260925 15:19:45.471282 17210 raft_consensus.cc:348] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.471552 17210 raft_consensus.cc:374] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:45.471652 17210 raft_consensus.cc:729] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38da62da3ec4492bb1ada9e416387d3, State: Initialized, Role: FOLLOWER
I20260925 15:19:45.471936 17210 consensus_queue.cc:260] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.472522 17210 ts_tablet_manager.cc:1428] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:45.630932 17216 raft_consensus.cc:482] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:45.631168 17216 raft_consensus.cc:504] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.632864 17216 leader_election.cc:288] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857), 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325)
I20260925 15:19:45.640172 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.633505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:45295 (local address 127.8.226.66:35325)
0925 15:19:45.633660 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.633666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.633827 (+ 161us) server_negotiation.cc:407] Connection header received
0925 15:19:45.634157 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.634164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.634249 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.634332 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:45.634866 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.635704 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.637835 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.639047 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.639208 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.639280 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.639753 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.639891 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:19:45.640561 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.633188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:43845 (local address 127.8.226.65:37857)
0925 15:19:45.633397 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.633403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.634308 (+ 905us) server_negotiation.cc:407] Connection header received
0925 15:19:45.634484 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.634489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.634552 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.634599 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:45.635338 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.636503 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.638505 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.639457 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.639646 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.639720 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.639984 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.640111 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:19:45.640501 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.633132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37857 (local address 127.8.226.67:43845)
0925 15:19:45.634206 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.634221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.634335 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.634734 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.634737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.634753 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:45.635156 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.635168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.636650 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.636655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.638366 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.638375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.639684 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.639690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.639841 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.639863 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.639962 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.640080 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":1,"thread_start_us":408,"threads_started":1}
I20260925 15:19:45.640901 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.633108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:35325 (local address 127.8.226.67:45295)
0925 15:19:45.633723 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.633741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.633959 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.634398 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.634402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.634427 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:45.634662 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.634673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.635831 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.635837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.637669 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.637683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.639363 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.639369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.639550 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.639589 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.640337 (+ 748us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.640473 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":47,"thread_start_us":804,"threads_started":1}
I20260925 15:19:45.641637 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64920551545f4b5d9bd62250b3e104eb" candidate_uuid: "f0147c8ca41c4193bb7176ad117d08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38da62da3ec4492bb1ada9e416387d3" is_pre_election: true
I20260925 15:19:45.641814 16996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64920551545f4b5d9bd62250b3e104eb" candidate_uuid: "f0147c8ca41c4193bb7176ad117d08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" is_pre_election: true
I20260925 15:19:45.641906 16862 raft_consensus.cc:2454] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0147c8ca41c4193bb7176ad117d08da in term 0.
I20260925 15:19:45.642052 16996 raft_consensus.cc:2454] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0147c8ca41c4193bb7176ad117d08da in term 0.
I20260925 15:19:45.642414 17068 leader_election.cc:302] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d38da62da3ec4492bb1ada9e416387d3, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:45.642824 17216 raft_consensus.cc:2781] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:45.642936 17216 raft_consensus.cc:482] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:45.643049 17216 raft_consensus.cc:3037] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.645145 17216 raft_consensus.cc:504] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.645797 17216 leader_election.cc:288] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [CANDIDATE]: Term 1 election: Requested vote from peers d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857), 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325)
I20260925 15:19:45.646174 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64920551545f4b5d9bd62250b3e104eb" candidate_uuid: "f0147c8ca41c4193bb7176ad117d08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38da62da3ec4492bb1ada9e416387d3"
I20260925 15:19:45.646443 16862 raft_consensus.cc:3037] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.646433 16996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64920551545f4b5d9bd62250b3e104eb" candidate_uuid: "f0147c8ca41c4193bb7176ad117d08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7"
I20260925 15:19:45.646636 16996 raft_consensus.cc:3037] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:45.648865 16862 raft_consensus.cc:2454] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0147c8ca41c4193bb7176ad117d08da in term 1.
I20260925 15:19:45.649294 17068 leader_election.cc:302] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d38da62da3ec4492bb1ada9e416387d3, f0147c8ca41c4193bb7176ad117d08da; no voters:
I20260925 15:19:45.649668 17216 raft_consensus.cc:2781] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:45.649919 16996 raft_consensus.cc:2454] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0147c8ca41c4193bb7176ad117d08da in term 1.
I20260925 15:19:45.658665 17216 raft_consensus.cc:686] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 1 LEADER]: Becoming Leader. State: Replica: f0147c8ca41c4193bb7176ad117d08da, State: Running, Role: LEADER
I20260925 15:19:45.659008 17216 consensus_queue.cc:237] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } }
I20260925 15:19:45.662442 16699 catalog_manager.cc:5495] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da reported cstate change: term changed from 0 to 1, leader changed from <none> to f0147c8ca41c4193bb7176ad117d08da (127.8.226.67). New cstate: current_term: 1 leader_uuid: "f0147c8ca41c4193bb7176ad117d08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0147c8ca41c4193bb7176ad117d08da" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40725 } health_report { overall_health: HEALTHY } } }
I20260925 15:19:45.713218 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.707561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.8.226.67:40725)
0925 15:19:45.708145 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.708149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.708165 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:19:45.708238 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.708241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.708316 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.708484 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.708880 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.709687 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.710984 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.711422 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.711569 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.711984 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.712541 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.712719 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.712797 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.712871 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":324,"threads_started":1}
I20260925 15:19:45.728600 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.722910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:37857 (local address 127.8.226.67:46129)
0925 15:19:45.723505 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.723519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.723622 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.723947 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.723950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.723958 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:45.724190 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.724205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.725665 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.725668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.726640 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.726649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.727607 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.727611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.727718 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.727762 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.728172 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.728178 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.728225 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.728319 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":300,"threads_started":1}
I20260925 15:19:45.728648 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.722909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:46129 (local address 127.8.226.65:37857)
0925 15:19:45.723055 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.723059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.723605 (+ 546us) server_negotiation.cc:407] Connection header received
0925 15:19:45.723747 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.723753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.723837 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.723932 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.724336 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.725535 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.726769 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.727439 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.727564 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.727941 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.728086 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.728201 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.728253 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.728326 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:19:45.737322 16996 raft_consensus.cc:1261] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Refusing update from remote peer f0147c8ca41c4193bb7176ad117d08da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.737352 16862 raft_consensus.cc:1261] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Refusing update from remote peer f0147c8ca41c4193bb7176ad117d08da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:45.737910 17293 consensus_queue.cc:1035] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ce821999cb4f4aa8bf48d6024d6ad7" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.738474 17216 consensus_queue.cc:1035] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38da62da3ec4492bb1ada9e416387d3" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:45.779323 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.772569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:54609 (local address 127.8.226.67:40725)
0925 15:19:45.772722 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.772728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.773146 (+ 418us) server_negotiation.cc:407] Connection header received
0925 15:19:45.773321 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.773326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.773404 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.773494 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:45.773956 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.775064 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.777285 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.777967 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.778095 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.778570 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.778713 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.778806 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.778949 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.779026 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:19:45.779649 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.772494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40725 (local address 127.8.226.65:54609)
0925 15:19:45.773055 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:45.773073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:45.773182 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:45.773517 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:45.773521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:45.773533 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:19:45.773777 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.773790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.775220 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.775226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.777092 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:45.777107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.778147 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.778153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:45.778345 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.778389 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:19:45.778823 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:19:45.778837 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:19:45.778917 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:19:45.779051 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":286,"threads_started":1}
I20260925 15:19:45.812354 16697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:45.812599 16697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:45.815588 16697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e84da461e39a4670964776f35a6f2fbc: Sending DeleteTablet for 3 replicas of tablet 64920551545f4b5d9bd62250b3e104eb
I20260925 15:19:45.816550 17114 tablet_service.cc:1514] Processing DeleteTablet for tablet 64920551545f4b5d9bd62250b3e104eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:51152
I20260925 15:19:45.816699 16975 tablet_service.cc:1514] Processing DeleteTablet for tablet 64920551545f4b5d9bd62250b3e104eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:47424
I20260925 15:19:45.816715 16841 tablet_service.cc:1514] Processing DeleteTablet for tablet 64920551545f4b5d9bd62250b3e104eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:45 UTC) from {username='oryx', principal='oryx/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:34454
I20260925 15:19:45.816824 17233 tablet_replica.cc:330] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: stopping tablet replica
I20260925 15:19:45.817008 17234 tablet_replica.cc:330] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: stopping tablet replica
I20260925 15:19:45.817021 17235 tablet_replica.cc:330] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: stopping tablet replica
I20260925 15:19:45.817052 17233 raft_consensus.cc:2229] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:45.817227 17234 raft_consensus.cc:2229] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:45.817231 17235 raft_consensus.cc:2229] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:45.817408 17235 raft_consensus.cc:2258] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.817427 17234 raft_consensus.cc:2258] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.817518 17233 raft_consensus.cc:2258] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:45.818840 17235 ts_tablet_manager.cc:1905] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.818707 17233 ts_tablet_manager.cc:1905] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.821918 17234 ts_tablet_manager.cc:1905] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:45.825486 17233 ts_tablet_manager.cc:1918] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:45.825632 17233 log.cc:1196] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/64920551545f4b5d9bd62250b3e104eb
I20260925 15:19:45.826037 17233 ts_tablet_manager.cc:1939] T 64920551545f4b5d9bd62250b3e104eb P f0147c8ca41c4193bb7176ad117d08da: Deleting consensus metadata
I20260925 15:19:45.827031 16685 catalog_manager.cc:4841] TS f0147c8ca41c4193bb7176ad117d08da (127.8.226.67:40725): tablet 64920551545f4b5d9bd62250b3e104eb (table test-table [id=544014898101483591ecab0df47ca40c]) successfully deleted
I20260925 15:19:45.829074 17235 ts_tablet_manager.cc:1918] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:45.829236 17235 log.cc:1196] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/64920551545f4b5d9bd62250b3e104eb
I20260925 15:19:45.829625 17235 ts_tablet_manager.cc:1939] T 64920551545f4b5d9bd62250b3e104eb P 57ce821999cb4f4aa8bf48d6024d6ad7: Deleting consensus metadata
I20260925 15:19:45.830543 16684 catalog_manager.cc:4841] TS 57ce821999cb4f4aa8bf48d6024d6ad7 (127.8.226.66:35325): tablet 64920551545f4b5d9bd62250b3e104eb (table test-table [id=544014898101483591ecab0df47ca40c]) successfully deleted
I20260925 15:19:45.831887 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 16771
I20260925 15:19:45.837395 17234 ts_tablet_manager.cc:1918] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:19:45.837549 17234 log.cc:1196] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/64920551545f4b5d9bd62250b3e104eb
I20260925 15:19:45.837993 17234 ts_tablet_manager.cc:1939] T 64920551545f4b5d9bd62250b3e104eb P d38da62da3ec4492bb1ada9e416387d3: Deleting consensus metadata
I20260925 15:19:45.839020 16686 catalog_manager.cc:4841] TS d38da62da3ec4492bb1ada9e416387d3 (127.8.226.65:37857): tablet 64920551545f4b5d9bd62250b3e104eb (table test-table [id=544014898101483591ecab0df47ca40c]) successfully deleted
Sep 25 15:19:45 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.65@KRBTEST.COM
I20260925 15:19:45.852622 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:45.833035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.8.226.65:37857)
0925 15:19:45.833191 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:45.833208 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:45.833301 (+ 93us) server_negotiation.cc:407] Connection header received
0925 15:19:45.833526 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:45.833531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:45.833600 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:45.833676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:45.835307 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.836431 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.838467 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:45.839167 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:45.839268 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:45.845459 (+ 6191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:45.845473 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:45.845490 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:45.845527 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:45.849403 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:45.850240 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:45.850249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:45.850253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:45.850403 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:45.850807 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:45.850812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:45.850815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:45.851263 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:45.851537 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:45.852069 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 15:19:45.852267 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:19:45.854265 16902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:46.009315 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 16771
I20260925 15:19:46.034287 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 16911
Sep 25 15:19:46 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.66@KRBTEST.COM
I20260925 15:19:46.051241 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:46.035115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.8.226.66:35325)
0925 15:19:46.035308 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:46.035316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:46.035564 (+ 248us) server_negotiation.cc:407] Connection header received
0925 15:19:46.035721 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:46.035727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:46.035808 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:46.035934 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:46.037037 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.038152 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.039307 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.039940 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.040071 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:46.044599 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:46.044614 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:46.044631 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:46.044676 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:46.048272 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:46.049101 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:46.049109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:46.049112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:46.049252 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:46.049619 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:46.049625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:46.049630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:46.050056 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:46.050349 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:46.050637 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:19:46.050874 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:19:46.051919 17041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:46.172832 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 16911
I20260925 15:19:46.194356 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17050
Sep 25 15:19:46 dist-test-slave-8vw7 krb5kdc[16655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.226.67@KRBTEST.COM
I20260925 15:19:46.210074 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:46.195236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.8.226.67:40725)
0925 15:19:46.195460 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:46.195466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:46.195485 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:19:46.195602 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:46.195607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:46.195663 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:46.195781 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:46.197017 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.197768 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.199035 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.199483 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.199600 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:46.204684 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:46.204695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:46.204720 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:46.204750 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:46.207568 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:46.208260 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:46.208266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:46.208268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:46.208374 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:46.208713 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:46.208715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:46.208717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:46.209087 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:46.209294 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:46.209624 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:19:46.209777 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:19:46.210780 17180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:46.334843 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17050
I20260925 15:19:46.356498 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 16668
I20260925 15:19:46.357666 16740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:46.470965 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 16668
2026-09-25T15:19:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6339 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:46 dist-test-slave-8vw7 krb5kdc[17313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:46 dist-test-slave-8vw7 krb5kdc[17313](info): set up 2 sockets
Sep 25 15:19:46 dist-test-slave-8vw7 krb5kdc[17313](info): commencing operation
krb5kdc: starting...
W20260925 15:19:47.514222 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:47 dist-test-slave-8vw7 krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:47.715655 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35371
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:35371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:48.018574 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:48.018886 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:48.019044 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:48.026129 17326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:48.026279 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:48.026382 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:48.026481 17326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:48.026571 17326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:48.036152 17326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:35371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:48.037456 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:48.038681 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:48.049840 17331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:48.049885 17332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:48.049799 17334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:48.054188 17326 server_base.cc:1034] running on GCE node
I20260925 15:19:48.054865 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:48.056149 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:48.057337 17326 hybrid_clock.cc:648] HybridClock initialized: now 1790349588057299 us; error 32 us; skew 500 ppm
I20260925 15:19:48.057646 17326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:19:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1650 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:48 dist-test-slave-8vw7 krb5kdc[17340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:48 dist-test-slave-8vw7 krb5kdc[17340](info): set up 2 sockets
Sep 25 15:19:48 dist-test-slave-8vw7 krb5kdc[17340](info): commencing operation
krb5kdc: starting...
W20260925 15:19:49.202097 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:19:49 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:19:49.399981 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38373
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:49.703682 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:49.703997 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:49.704155 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:49.711102 17353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:49.711220 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:49.711314 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:49.711447 17353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:49.711527 17353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:49.720579 17353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:49.721820 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:49.723563 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:49.734691 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:49.734766 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:49.734748 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:49.735528 17353 server_base.cc:1034] running on GCE node
I20260925 15:19:49.736248 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:49.737581 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:49.738792 17353 hybrid_clock.cc:648] HybridClock initialized: now 1790349589738756 us; error 47 us; skew 500 ppm
I20260925 15:19:49.739080 17353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:49 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:49.746855 17353 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:49.748283 17353 webserver.cc:460] Webserver started at http://127.8.226.126:40425/ using document root <none> and password file <none>
I20260925 15:19:49.748924 17353 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:49.749104 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:49.749404 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:49.752209 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "1663e01e7778492d8e1e40fee0e880ff"
format_stamp: "Formatted at 2026-09-25 15:19:49 on dist-test-slave-8vw7"
server_key: "f7b06d09c6f7719f9d443a51729b668a"
server_key_iv: "d8c6ed5186377acd409dd7e25ac9f7d1"
server_key_version: "encryptionkey@0"
I20260925 15:19:49.753033 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "1663e01e7778492d8e1e40fee0e880ff"
format_stamp: "Formatted at 2026-09-25 15:19:49 on dist-test-slave-8vw7"
server_key: "f7b06d09c6f7719f9d443a51729b668a"
server_key_iv: "d8c6ed5186377acd409dd7e25ac9f7d1"
server_key_version: "encryptionkey@0"
I20260925 15:19:49.757752 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:49.761023 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:49.762254 17353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:19:49.762422 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "1663e01e7778492d8e1e40fee0e880ff"
format_stamp: "Formatted at 2026-09-25 15:19:49 on dist-test-slave-8vw7"
server_key: "f7b06d09c6f7719f9d443a51729b668a"
server_key_iv: "d8c6ed5186377acd409dd7e25ac9f7d1"
server_key_version: "encryptionkey@0"
I20260925 15:19:49.762614 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:49.775075 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:49.778913 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:49.779150 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:49.796344 17353 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:38373
I20260925 15:19:49.796361 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:38373 every 8 connection(s)
I20260925 15:19:49.797578 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:49.797969 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17353
I20260925 15:19:49.798199 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:19:49.799026 9097 external_mini_cluster.cc:1428] Setting key dd9a4723ecdd5bb5b76e107b58b14ca0
I20260925 15:19:49.801654 17420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:19:49 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:49.811996 17420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Bootstrap starting.
I20260925 15:19:49.815629 17420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:49.816738 17420 log.cc:824] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:49.819703 17420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: No bootstrap required, opened a new log
I20260925 15:19:49.820012 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:49.800828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.8.226.126:38373)
0925 15:19:49.801542 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:49.801553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:49.801589 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:19:49.803157 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:49.803178 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:49.803506 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:49.803887 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:49.805316 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:49.806550 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:49.807650 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:49.808224 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:49.808431 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:49.812581 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:49.812590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:49.812606 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:49.812652 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:49.816157 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:49.816723 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:49.816733 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:49.816741 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:49.816943 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:49.817363 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:49.817370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:49.817373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:49.817815 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:49.818113 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:49.818480 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:19:49.818842 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":342,"threads_started":1}
I20260925 15:19:49.826992 17420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:49.827411 17420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:49.827509 17420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1663e01e7778492d8e1e40fee0e880ff, State: Initialized, Role: FOLLOWER
I20260925 15:19:49.827953 17420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:49.828176 17420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:49.828347 17420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:49.828505 17420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:49.830572 17420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:49.831067 17420 leader_election.cc:302] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1663e01e7778492d8e1e40fee0e880ff; no voters:
I20260925 15:19:49.831615 17420 leader_election.cc:288] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:49.831730 17425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:49.832393 17425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 LEADER]: Becoming Leader. State: Replica: 1663e01e7778492d8e1e40fee0e880ff, State: Running, Role: LEADER
I20260925 15:19:49.833003 17425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:49.833669 17420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:49.835536 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1663e01e7778492d8e1e40fee0e880ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } } }
I20260925 15:19:49.835646 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1663e01e7778492d8e1e40fee0e880ff. Latest consensus state: current_term: 1 leader_uuid: "1663e01e7778492d8e1e40fee0e880ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } } }
I20260925 15:19:49.835882 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:49.835893 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:49.837878 17433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:49.842777 17433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:49.851217 17433 catalog_manager.cc:1347] Generated new cluster ID: be90e12e2fe84b3982312b836375c60c
I20260925 15:19:49.851305 17433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:49.872751 17433 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:49.874027 17433 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:49.879416 17433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Generated new TSK 0
I20260925 15:19:49.880172 17433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:19:49.932351 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38373
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:50.245864 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:50.246095 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:50.246244 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:50.252847 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:50.252950 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:50.253154 17446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:50.262041 17446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:50.263224 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:50.264410 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:50.276358 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:50.276413 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:50.276432 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:50.277101 17446 server_base.cc:1034] running on GCE node
I20260925 15:19:50.277575 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:50.278340 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:50.279515 17446 hybrid_clock.cc:648] HybridClock initialized: now 1790349590279479 us; error 33 us; skew 500 ppm
I20260925 15:19:50.279798 17446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:50 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:50.282716 17446 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:19:50.283967 17446 webserver.cc:460] Webserver started at http://127.8.226.65:34571/ using document root <none> and password file <none>
I20260925 15:19:50.284545 17446 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:50.284636 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:50.284864 17446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:50.287361 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "dc33861f658944a8afb3b61a5c30fbeb"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "4ffa30df3b82790681e85278c06ec21c"
server_key_iv: "b5c793c85beb02185a77ed0b03ed6cea"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.288015 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "dc33861f658944a8afb3b61a5c30fbeb"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "4ffa30df3b82790681e85278c06ec21c"
server_key_iv: "b5c793c85beb02185a77ed0b03ed6cea"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.292680 17446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 15:19:50.295641 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.296816 17446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:50.297029 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "dc33861f658944a8afb3b61a5c30fbeb"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "4ffa30df3b82790681e85278c06ec21c"
server_key_iv: "b5c793c85beb02185a77ed0b03ed6cea"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.297215 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:50.310842 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:50.314257 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:50.314504 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:50.315282 17446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:50.316373 17446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:50.316464 17446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.316599 17446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:50.316664 17446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.347896 17446 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:35013
I20260925 15:19:50.347935 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:35013 every 8 connection(s)
I20260925 15:19:50.349128 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:50.352031 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17446
I20260925 15:19:50.352236 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:50.353085 9097 external_mini_cluster.cc:1428] Setting key 65d01af511a8532cabc27852ea44e836
Sep 25 15:19:50 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:50.370702 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:50.351308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:35459 (local address 127.8.226.126:38373)
0925 15:19:50.351745 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:50.351752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:50.352909 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 15:19:50.355060 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:50.355064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:50.355131 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:50.355214 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:50.357136 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.357818 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.360163 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.360578 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.360659 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:50.364258 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:50.364266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:50.364273 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:50.364314 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:50.367696 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:50.368401 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:50.368408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:50.368414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:50.368549 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:50.369066 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:50.369073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:50.369077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:50.369338 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:50.369482 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:50.370130 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 15:19:50.370352 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260925 15:19:50.371536 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:50.351881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:35459)
0925 15:19:50.352724 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:50.352772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:50.354784 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:50.355472 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:50.355483 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:50.356177 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:50.356930 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:50.357002 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.357969 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.357977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:50.360026 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:50.360045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.360773 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.360781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:50.361091 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:50.361796 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:50.361829 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:50.364060 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:50.367835 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:50.367841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:50.367851 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:50.368278 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:50.368658 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:50.368665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:50.368669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:50.368909 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:50.369458 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:50.369464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:50.369793 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 15:19:50.370026 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:19:50.370486 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":344,"threads_started":1}
I20260925 15:19:50.373168 17574 heartbeater.cc:347] Connected to a master server at 127.8.226.126:38373
I20260925 15:19:50.373454 17574 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:50.374111 17574 heartbeater.cc:511] Master 127.8.226.126:38373 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:19:50.376273 17385 ts_manager.cc:194] Registered new tserver with Master: dc33861f658944a8afb3b61a5c30fbeb (127.8.226.65:35013)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
I20260925 15:19:50.377992 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:35459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:19:50.401393 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38373
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:50.695822 17580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:50.696058 17580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:50.696242 17580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:50.702942 17580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:50.703061 17580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:50.703284 17580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:50.712294 17580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:50.713454 17580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:50.714661 17580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:50.726951 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:50.726981 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:50.727025 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:50.727722 17580 server_base.cc:1034] running on GCE node
I20260925 15:19:50.728297 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:50.729301 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:50.730484 17580 hybrid_clock.cc:648] HybridClock initialized: now 1790349590730443 us; error 37 us; skew 500 ppm
I20260925 15:19:50.730772 17580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:50 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:50.733747 17580 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:19:50.734969 17580 webserver.cc:460] Webserver started at http://127.8.226.66:38879/ using document root <none> and password file <none>
I20260925 15:19:50.735545 17580 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:50.735630 17580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:50.735872 17580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:50.738547 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "f87e6c0f51194257932acddda68c866d"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "9298653b433629acf1c8508a0ca4dac7"
server_key_iv: "ac5d1a1a97b91bdc63b16719b45aa10f"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.739230 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "f87e6c0f51194257932acddda68c866d"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "9298653b433629acf1c8508a0ca4dac7"
server_key_iv: "ac5d1a1a97b91bdc63b16719b45aa10f"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.743810 17580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:19:50.746790 17595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.747849 17580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:50.747980 17580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "f87e6c0f51194257932acddda68c866d"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "9298653b433629acf1c8508a0ca4dac7"
server_key_iv: "ac5d1a1a97b91bdc63b16719b45aa10f"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.748155 17580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:50.780385 17580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:50.784366 17580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:50.784669 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:50.785573 17580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:50.786753 17580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:50.786816 17580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.786980 17580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:50.787024 17580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.822841 17580 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:33375
I20260925 15:19:50.822908 17707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:33375 every 8 connection(s)
I20260925 15:19:50.823980 17580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:50.829879 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17580
I20260925 15:19:50.830031 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:50.830457 9097 external_mini_cluster.cc:1428] Setting key b8b24f11691c0386dbe27aa0268ef0ed
Sep 25 15:19:50 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:50.843454 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:50.826200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:38435 (local address 127.8.226.126:38373)
0925 15:19:50.826362 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:50.826367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:50.827440 (+ 1073us) server_negotiation.cc:407] Connection header received
0925 15:19:50.829096 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:50.829102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:50.829201 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:50.829327 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:50.830854 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.831645 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.833602 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.834365 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.834493 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:50.837711 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:50.837719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:50.837725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:50.837782 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:50.840543 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:50.841411 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:50.841414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:50.841416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:50.841548 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:50.841988 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:50.841991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:50.841992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:50.842214 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:50.842324 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:50.843086 (+ 762us) server_negotiation.cc:299] Negotiation successful
0925 15:19:50.843229 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:19:50.844872 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:50.826661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:38435)
0925 15:19:50.827285 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:50.827324 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:50.828848 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:50.829512 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:50.829520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:50.830110 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:50.830694 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:50.830714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.831788 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.831793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:50.833420 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:50.833431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.834583 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.834588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:50.834831 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:50.835461 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:50.835486 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:50.837488 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:50.840681 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:50.840692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:50.840709 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:50.841308 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:50.841648 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:50.841656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:50.841662 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:50.841884 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:50.842344 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:50.842354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:50.842757 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 15:19:50.843019 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:19:50.843524 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":254,"threads_started":1}
I20260925 15:19:50.846128 17708 heartbeater.cc:347] Connected to a master server at 127.8.226.126:38373
I20260925 15:19:50.846411 17708 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:50.847045 17708 heartbeater.cc:511] Master 127.8.226.126:38373 requested a full tablet report, sending...
I20260925 15:19:50.848302 17385 ts_manager.cc:194] Registered new tserver with Master: f87e6c0f51194257932acddda68c866d (127.8.226.66:33375)
I20260925 15:19:50.849097 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:38435
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:19:50.876302 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38373
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:51.174394 17713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:51.174713 17713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:51.174880 17713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:51.181742 17713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:51.181874 17713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:51.182049 17713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:51.190948 17713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:51.192133 17713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:51.193399 17713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:51.205639 17718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:51.205597 17719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:51.205574 17721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:51.206377 17713 server_base.cc:1034] running on GCE node
I20260925 15:19:51.206842 17713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:51.207574 17713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:51.208804 17713 hybrid_clock.cc:648] HybridClock initialized: now 1790349591208752 us; error 41 us; skew 500 ppm
I20260925 15:19:51.209139 17713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:51 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:51.212016 17713 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:19:51.214123 17713 webserver.cc:460] Webserver started at http://127.8.226.67:38049/ using document root <none> and password file <none>
I20260925 15:19:51.214727 17713 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:51.214851 17713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:51.215127 17713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:51.217646 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "116083078710422486f338be49a4f809"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-8vw7"
server_key: "874db51aac6316b875e1d43cf8f69679"
server_key_iv: "64ce517e87da1cbea2a4c39dac7f9f24"
server_key_version: "encryptionkey@0"
I20260925 15:19:51.218330 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "116083078710422486f338be49a4f809"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-8vw7"
server_key: "874db51aac6316b875e1d43cf8f69679"
server_key_iv: "64ce517e87da1cbea2a4c39dac7f9f24"
server_key_version: "encryptionkey@0"
I20260925 15:19:51.222846 17713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:19:51.225788 17728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:51.226874 17713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:51.227046 17713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "116083078710422486f338be49a4f809"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-8vw7"
server_key: "874db51aac6316b875e1d43cf8f69679"
server_key_iv: "64ce517e87da1cbea2a4c39dac7f9f24"
server_key_version: "encryptionkey@0"
I20260925 15:19:51.227224 17713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:51.246639 17713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:51.250409 17713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:51.250689 17713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:51.251602 17713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:51.252846 17713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:51.252940 17713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:51.253110 17713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:51.253180 17713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:51.283774 17713 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:44995
I20260925 15:19:51.283849 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:44995 every 8 connection(s)
I20260925 15:19:51.284889 17713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:51.292572 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17713
I20260925 15:19:51.292723 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:51.293224 9097 external_mini_cluster.cc:1428] Setting key ad679f3086493c925fcbfe16d2dcbc53
Sep 25 15:19:51 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:19:51.302911 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.287071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:38543 (local address 127.8.226.126:38373)
0925 15:19:51.287219 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:51.287223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:51.288180 (+ 957us) server_negotiation.cc:407] Connection header received
0925 15:19:51.289961 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:51.289966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:51.290051 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:51.290173 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:51.291519 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.292276 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.294076 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.294678 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.294824 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:51.298013 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:51.298018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:51.298021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:51.298047 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:51.300414 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.301063 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.301066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.301068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.301172 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.301601 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.301604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.301606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.301845 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:51.301948 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:51.302498 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 15:19:51.302674 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:19:51.303694 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.287433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:38543)
0925 15:19:51.288020 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:51.288072 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:51.289720 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:51.290388 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:51.290396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:51.290850 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:51.291360 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:51.291378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.292434 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.292438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:51.293940 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:51.293951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.294912 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.294920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:51.295154 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:51.295781 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:51.295806 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:51.297850 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:51.300518 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:51.300524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:51.300533 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:51.300926 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:51.301255 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:51.301258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:51.301260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:51.301477 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:51.301942 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:51.301947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:51.302206 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 15:19:51.302401 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 15:19:51.302736 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":220,"threads_started":1}
I20260925 15:19:51.304919 17841 heartbeater.cc:347] Connected to a master server at 127.8.226.126:38373
I20260925 15:19:51.305233 17841 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:51.305924 17841 heartbeater.cc:511] Master 127.8.226.126:38373 requested a full tablet report, sending...
I20260925 15:19:51.307490 17385 ts_manager.cc:194] Registered new tserver with Master: 116083078710422486f338be49a4f809 (127.8.226.67:44995)
I20260925 15:19:51.308109 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:51.308387 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:38543
WARNING: no policy specified for oryx/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab.
Entry for principal oryx/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab.
WARNING: no policy specified for oryx/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.66@KRBTEST.COM" created.
I20260925 15:19:51.380831 17574 heartbeater.cc:503] Master 127.8.226.126:38373 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab.
Entry for principal oryx/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab.
WARNING: no policy specified for oryx/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab.
Entry for principal oryx/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab.
I20260925 15:19:51.419773 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17446
Sep 25 15:19:51 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:19:51.434360 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.420918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.8.226.65:35013)
0925 15:19:51.421509 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:51.421516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:51.421558 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:19:51.421628 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:51.421634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:51.421842 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:51.421969 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:51.423198 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.424068 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.425160 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.425674 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.425769 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:51.429333 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:51.429342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:51.429355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:51.429382 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:51.432081 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.432656 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.432662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.432664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.432788 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.433126 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.433129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.433130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.433413 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:51.433606 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:51.433776 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:19:51.434016 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":303,"threads_started":1}
I20260925 15:19:51.435614 17569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:51.514648 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17446
I20260925 15:19:51.529421 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17580
Sep 25 15:19:51 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:19:51.544196 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.530285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.8.226.66:33375)
0925 15:19:51.530892 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:51.530899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:51.530932 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:51.530985 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:51.530991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:51.531174 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:51.531361 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:51.532611 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.533670 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.534830 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.535338 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.535483 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:51.539347 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:51.539356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:51.539367 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:51.539400 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:51.542077 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.542628 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.542633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.542636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.542756 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.543019 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.543021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.543023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.543345 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:51.543533 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:51.543746 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:19:51.543918 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":349,"threads_started":1}
I20260925 15:19:51.545411 17703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:51.622911 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17580
I20260925 15:19:51.637686 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17713
Sep 25 15:19:51 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:19:51.653712 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.638573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.8.226.67:44995)
0925 15:19:51.639144 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:51.639151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:51.639178 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:19:51.639236 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:51.639243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:51.639481 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:51.639602 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:51.640704 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.641701 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.642815 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:51.643312 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:51.643401 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:51.648227 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:51.648238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:51.648251 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:51.648279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:51.650994 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.651687 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.651695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.651697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.651812 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:51.652212 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:51.652219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:51.652222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:51.652533 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:51.652872 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:51.653133 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 15:19:51.653373 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
I20260925 15:19:51.654898 17836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:51.732292 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17713
I20260925 15:19:51.747412 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17353
I20260925 15:19:51.748629 17415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:19:51.847227 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17353
I20260925 15:19:51.862615 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38373
--webserver_interface=127.8.226.126
--webserver_port=40425
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:52.163022 17858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:52.163268 17858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:52.163347 17858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:52.169588 17858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:52.169715 17858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:52.169811 17858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:52.169896 17858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:52.169940 17858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:52.178499 17858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=40425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:52.179533 17858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:52.180693 17858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:52.191655 17864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:52.191628 17866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:52.191752 17863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:52.192195 17858 server_base.cc:1034] running on GCE node
I20260925 15:19:52.193028 17858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:52.194267 17858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:52.195451 17858 hybrid_clock.cc:648] HybridClock initialized: now 1790349592195417 us; error 23 us; skew 500 ppm
I20260925 15:19:52.195762 17858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:52 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:52.198541 17858 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:19:52.199898 17858 webserver.cc:460] Webserver started at http://127.8.226.126:40425/ using document root <none> and password file <none>
I20260925 15:19:52.200541 17858 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:52.200690 17858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:52.205484 17858 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:19:52.207902 17873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:52.209165 17858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:52.209339 17858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "1663e01e7778492d8e1e40fee0e880ff"
format_stamp: "Formatted at 2026-09-25 15:19:49 on dist-test-slave-8vw7"
server_key: "f7b06d09c6f7719f9d443a51729b668a"
server_key_iv: "d8c6ed5186377acd409dd7e25ac9f7d1"
server_key_version: "encryptionkey@0"
I20260925 15:19:52.210482 17858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:52.229388 17858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:52.233487 17858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:52.233691 17858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:52.250771 17858 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:38373
I20260925 15:19:52.250800 17924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:38373 every 8 connection(s)
I20260925 15:19:52.252060 17858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:52.258100 17925 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:19:52.261051 17925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Bootstrap starting.
I20260925 15:19:52.261488 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17858
I20260925 15:19:52.262270 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:35013
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:38373
--webserver_port=34571
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:52.278837 17925 log.cc:824] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:52.286829 17925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:19:52.287307 17925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Bootstrap complete.
I20260925 15:19:52.295774 17925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:52.296535 17925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1663e01e7778492d8e1e40fee0e880ff, State: Initialized, Role: FOLLOWER
I20260925 15:19:52.297156 17925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:52.297380 17925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:52.297504 17925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:52.297675 17925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:19:52.301088 17925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:52.301528 17925 leader_election.cc:302] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1663e01e7778492d8e1e40fee0e880ff; no voters:
I20260925 15:19:52.302165 17925 leader_election.cc:288] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:19:52.302304 17929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:19:52.303550 17929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [term 2 LEADER]: Becoming Leader. State: Replica: 1663e01e7778492d8e1e40fee0e880ff, State: Running, Role: LEADER
I20260925 15:19:52.304081 17929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } }
I20260925 15:19:52.304277 17925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:52.306397 17931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1663e01e7778492d8e1e40fee0e880ff. Latest consensus state: current_term: 2 leader_uuid: "1663e01e7778492d8e1e40fee0e880ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } } }
I20260925 15:19:52.306751 17931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:52.306556 17930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1663e01e7778492d8e1e40fee0e880ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1663e01e7778492d8e1e40fee0e880ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38373 } } }
I20260925 15:19:52.306977 17930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:52.313460 17936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:52.319931 17936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:52.321707 17936 catalog_manager.cc:1259] Loaded cluster ID: be90e12e2fe84b3982312b836375c60c
I20260925 15:19:52.321867 17936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:52.326781 17936 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:52.328523 17936 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1663e01e7778492d8e1e40fee0e880ff: Loaded TSK: 0
I20260925 15:19:52.330506 17936 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:19:52.599653 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:52.599954 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:52.600108 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:52.607401 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:52.607542 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:52.607710 17927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:19:52.617357 17927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.65.keytab
--principal=oryx/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:35013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=34571
--tserver_master_addrs=127.8.226.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:52.618593 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:52.619871 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:52.633610 17951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:52.633579 17952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:52.634037 17954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:52.634653 17927 server_base.cc:1034] running on GCE node
I20260925 15:19:52.635193 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:52.636025 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:52.637233 17927 hybrid_clock.cc:648] HybridClock initialized: now 1790349592637182 us; error 46 us; skew 500 ppm
I20260925 15:19:52.637532 17927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:52 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349592, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:52.640381 17927 init.cc:377] Logged in from keytab as oryx/127.8.226.65@KRBTEST.COM (short username oryx)
I20260925 15:19:52.641947 17927 webserver.cc:460] Webserver started at http://127.8.226.65:34571/ using document root <none> and password file <none>
I20260925 15:19:52.642573 17927 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:52.642709 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:52.647608 17927 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 15:19:52.651384 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:52.652743 17927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:52.652920 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "dc33861f658944a8afb3b61a5c30fbeb"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "4ffa30df3b82790681e85278c06ec21c"
server_key_iv: "b5c793c85beb02185a77ed0b03ed6cea"
server_key_version: "encryptionkey@0"
I20260925 15:19:52.653798 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:52.674449 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:52.677891 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:52.678148 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:52.678959 17927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:52.680073 17927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:52.680181 17927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:52.680297 17927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:52.680375 17927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:52.710402 17927 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:35013
I20260925 15:19:52.710492 18073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:35013 every 8 connection(s)
I20260925 15:19:52.711841 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:52.712771 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 17927
I20260925 15:19:52.713554 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:33375
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:38373
--webserver_port=38879
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:19:52 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:52 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:19:53.023898 18078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:53.024173 18078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:53.024323 18078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:53.030885 18078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:53.031006 18078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:53.031204 18078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:19:53.044770 18078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.66.keytab
--principal=oryx/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:33375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=38879
--tserver_master_addrs=127.8.226.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:53.046317 18078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:53.047566 18078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:53.064060 18084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:53.064496 18087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:53.064167 18085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:53.065529 18078 server_base.cc:1034] running on GCE node
I20260925 15:19:53.066515 18078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:53.067966 18078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:53.069311 18078 hybrid_clock.cc:648] HybridClock initialized: now 1790349593069233 us; error 46 us; skew 500 ppm
I20260925 15:19:53.069864 18078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:53 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349593, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:53.075042 18078 init.cc:377] Logged in from keytab as oryx/127.8.226.66@KRBTEST.COM (short username oryx)
I20260925 15:19:53.077126 18078 webserver.cc:460] Webserver started at http://127.8.226.66:38879/ using document root <none> and password file <none>
I20260925 15:19:53.078246 18078 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:53.078430 18078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:53.087303 18078 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:19:53.091435 18094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.093497 18078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:19:53.093709 18078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "f87e6c0f51194257932acddda68c866d"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-8vw7"
server_key: "9298653b433629acf1c8508a0ca4dac7"
server_key_iv: "ac5d1a1a97b91bdc63b16719b45aa10f"
server_key_version: "encryptionkey@0"
I20260925 15:19:53.095551 18078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:53.141893 18078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:53.145900 18078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:53.146180 18078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:53.147136 18078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:53.148435 18078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:53.148552 18078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.148684 18078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:53.148764 18078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.186429 18078 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:33375
I20260925 15:19:53.186602 18206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:33375 every 8 connection(s)
I20260925 15:19:53.187917 18078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:53.195778 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18078
I20260925 15:19:53.196432 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:44995
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:38373
--webserver_port=38049
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:19:53 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:53 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:19:53.514750 18211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:53.515007 18211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:53.515161 18211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:53.521849 18211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:53.521978 18211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:53.522161 18211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:19:53.530827 18211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/krb5kdc/oryx_127.8.226.67.keytab
--principal=oryx/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:44995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=38049
--tserver_master_addrs=127.8.226.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:19:53.532034 18211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:53.533298 18211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:53.549376 18218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:53.549481 18217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:53.555150 18211 server_base.cc:1034] running on GCE node
W20260925 15:19:53.555583 18220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:53.556239 18211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:53.557325 18211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:53.558573 18211 hybrid_clock.cc:648] HybridClock initialized: now 1790349593558516 us; error 53 us; skew 500 ppm
I20260925 15:19:53.558866 18211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:19:53 dist-test-slave-8vw7 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349593, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:53.561978 18211 init.cc:377] Logged in from keytab as oryx/127.8.226.67@KRBTEST.COM (short username oryx)
I20260925 15:19:53.566089 18211 webserver.cc:460] Webserver started at http://127.8.226.67:38049/ using document root <none> and password file <none>
I20260925 15:19:53.566716 18211 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:53.566861 18211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:53.572095 18211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:19:53.574798 18227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.576128 18211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:53.576285 18211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "116083078710422486f338be49a4f809"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-8vw7"
server_key: "874db51aac6316b875e1d43cf8f69679"
server_key_iv: "64ce517e87da1cbea2a4c39dac7f9f24"
server_key_version: "encryptionkey@0"
I20260925 15:19:53.577216 18211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:53.595538 18211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:53.599552 18211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:53.599817 18211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:53.600730 18211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:53.602412 18211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:53.602531 18211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.602697 18211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:53.602787 18211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.651342 18339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:44995 every 8 connection(s)
I20260925 15:19:53.652017 18211 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:44995
I20260925 15:19:53.654615 18211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:53.662216 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18211
Sep 25 15:19:53 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:53 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:53.695735 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:53.663187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.8.226.126:38373)
0925 15:19:53.675124 (+ 11937us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:53.675132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:53.678082 (+ 2950us) server_negotiation.cc:407] Connection header received
0925 15:19:53.678262 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:53.678267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:53.678333 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:53.678437 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:53.680201 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.681260 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.683216 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.683583 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.683718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:53.687119 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:53.687126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:53.687140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:53.687167 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:53.690242 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:53.691249 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:53.691258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:53.691273 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:53.691431 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:53.692743 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:53.692748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:53.692751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:53.693374 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:53.693697 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:53.694611 (+ 914us) server_negotiation.cc:299] Negotiation successful
0925 15:19:53.695051 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11797,"thread_start_us":11661,"threads_started":1}
W20260925 15:19:54.198203 18077 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.472s user 0.005s sys 0.011s
W20260925 15:19:54.199848 18076 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:52.714925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:33013 (local address 127.8.226.126:38373)
0925 15:19:52.715630 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:52.715642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:52.717810 (+ 2168us) server_negotiation.cc:407] Connection header received
0925 15:19:52.719818 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:52.719831 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:52.720130 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:52.720467 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:52.721978 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.723156 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.724630 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.725364 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.725557 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.199569 (+1474012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:33013: BlockingRecv error: recv got EOF from 127.8.226.65:33013 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":300,"threads_started":1}
I20260925 15:19:54.200562 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:52.715135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:33013)
0925 15:19:52.717633 (+ 2498us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:52.717675 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:52.719289 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:52.720597 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:52.720608 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:52.721246 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:52.721777 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:52.721794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.723311 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.723315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:52.724481 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:52.724492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.725601 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.725609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:52.725875 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:52.726527 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:52.726551 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.199350 (+1472799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2273,"thread_start_us":2122,"threads_started":1}
W20260925 15:19:54.202549 18074 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:38373 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.226.126:38373: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:19:54.221695 18076 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:54.204855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:42887 (local address 127.8.226.126:38373)
0925 15:19:54.205093 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.205102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.205605 (+ 503us) server_negotiation.cc:407] Connection header received
0925 15:19:54.205762 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.205769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.205909 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.206028 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.207525 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.209117 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.212481 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.213608 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.213791 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.221310 (+ 7519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:42887: BlockingRecv error: recv got EOF from 127.8.226.65:42887 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:19:54.221436 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.204380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:42887)
0925 15:19:54.205293 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:54.205318 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:54.205470 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:54.206289 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:54.206297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:54.207020 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:54.207308 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.207324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.209360 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.209369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.212252 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.212274 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.213850 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.213859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.214215 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.215150 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:54.215182 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.220999 (+ 5817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":746}
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:54.240645 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.223649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:46353)
0925 15:19:54.224413 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:54.224439 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:54.224681 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:54.225491 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:54.225498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:54.226233 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:54.226598 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.226612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.228649 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.228657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.231219 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.231238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.232890 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.232899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.233384 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.234302 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:54.234335 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.240193 (+ 5858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":600,"threads_started":1}
W20260925 15:19:54.240828 18076 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:54.224034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:46353 (local address 127.8.226.126:38373)
0925 15:19:54.224266 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.224275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.224545 (+ 270us) server_negotiation.cc:407] Connection header received
0925 15:19:54.224860 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.224869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.224996 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.225089 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.226820 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.228379 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.231423 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.232359 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.232488 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.240487 (+ 7999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:46353: BlockingRecv error: recv got EOF from 127.8.226.65:46353 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:19:54.241271 18074 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:19:54.568782 18210 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.365s user 0.000s sys 0.017s
W20260925 15:19:54.570251 18209 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:53.191302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:38079 (local address 127.8.226.126:38373)
0925 15:19:53.191935 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:53.191945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:53.192806 (+ 861us) server_negotiation.cc:407] Connection header received
0925 15:19:53.194707 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:53.194716 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:53.194841 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:53.195027 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:53.196903 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.198562 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.200874 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.201846 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.202021 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.569903 (+1367882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:38079: BlockingRecv error: recv got EOF from 127.8.226.66:38079 (error 108)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":306,"threads_started":1}
I20260925 15:19:54.570776 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:53.191858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:38079)
0925 15:19:53.192636 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:53.192687 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:53.194413 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:53.195201 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:53.195208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:53.195968 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:53.196716 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:53.196740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.198750 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.198758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:53.200719 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:53.200737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.202121 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.202131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:53.202475 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:53.203422 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:53.203454 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.569655 (+1366201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":323,"threads_started":1}
W20260925 15:19:54.571947 18207 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:38373 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.226.126:38373: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:54.583902 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.572725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:33503)
0925 15:19:54.573384 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:54.573409 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:54.573567 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:54.574049 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:54.574052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:54.574376 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:54.574530 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.574538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.576449 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.576452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.577451 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.577462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.578680 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.578684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.578800 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.579516 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:54.579532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.583713 (+ 4181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472}
W20260925 15:19:54.584748 18209 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:54.573002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:33503 (local address 127.8.226.126:38373)
0925 15:19:54.573206 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.573213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.573488 (+ 275us) server_negotiation.cc:407] Connection header received
0925 15:19:54.573763 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.573771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.573872 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.573991 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.574672 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.576304 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.577587 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.578509 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.578681 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.584393 (+ 5712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:33503: BlockingRecv error: recv got EOF from 127.8.226.66:33503 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:54.594489 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.584595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:46389)
0925 15:19:54.584946 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:54.584962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:54.585110 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:54.585544 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:54.585547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:54.585783 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:54.585930 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.585936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.587749 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.587752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.588626 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.588635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.589852 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.589856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.589992 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.590630 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:54.590645 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.594233 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260}
W20260925 15:19:54.594648 18076 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:54.584783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:46389 (local address 127.8.226.126:38373)
0925 15:19:54.584943 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.584948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.585037 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:19:54.585260 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.585268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.585386 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.585535 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.586076 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.587591 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.588752 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.589694 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.589867 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.594416 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:46389: BlockingRecv error: recv got EOF from 127.8.226.66:46389 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:19:54.594893 18207 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:19:54.759328 18342 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.068s user 0.001s sys 0.012s
W20260925 15:19:54.760659 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:53.661596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:38641 (local address 127.8.226.126:38373)
0925 15:19:53.669405 (+ 7809us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:53.669412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:53.674039 (+ 4627us) server_negotiation.cc:407] Connection header received
0925 15:19:53.674131 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:53.674138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:53.674259 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:53.674390 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:53.681544 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.682767 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.686133 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.686752 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.686845 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.760329 (+1073484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:38641: BlockingRecv error: recv got EOF from 127.8.226.67:38641 (error 108)
Metrics: {"server-negotiator.queue_time_us":7689,"thread_start_us":7778,"threads_started":1}
I20260925 15:19:54.761190 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:53.660917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:38641)
0925 15:19:53.670024 (+ 9107us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:53.670095 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:53.673578 (+ 3483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:53.677364 (+ 3786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:53.677379 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:53.679240 (+ 1861us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:53.680103 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:53.680138 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.682967 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.682973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:53.685940 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:53.685957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:53.688596 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:53.688609 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:53.690606 (+ 1997us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:53.691650 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:53.691677 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.760087 (+1068410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8640,"thread_start_us":8418,"threads_started":1}
W20260925 15:19:54.762003 18340 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:38373 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.226.126:38373: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:54.774839 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.762610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:44901)
0925 15:19:54.762801 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:54.762821 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:54.762916 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:54.763178 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:54.763181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:54.763482 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:54.763625 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.763633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.764605 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.764610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.765540 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.765552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.766197 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.766202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.766359 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.771347 (+ 4988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:54.771363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.774570 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:19:54.774947 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:54.762670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:44901 (local address 127.8.226.126:38373)
0925 15:19:54.762811 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.762815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.762880 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:19:54.763013 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.763017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.763069 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.763145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.763761 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.764467 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.765670 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.766036 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.766151 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.774741 (+ 8590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:44901: BlockingRecv error: recv got EOF from 127.8.226.67:44901 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:54 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:54.783066 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.775544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:34775)
0925 15:19:54.775688 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:54.775703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:54.775786 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:54.776026 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:54.776029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:54.776263 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:54.776425 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.776431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.777407 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.777411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.778276 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:54.778285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.778857 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.778861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:54.779024 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.779529 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:54.779544 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:54.782890 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:19:54.783310 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:54.775597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:34775 (local address 127.8.226.126:38373)
0925 15:19:54.775797 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.775800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.775833 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:54.775865 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.775868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.775925 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.775990 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.776544 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.777292 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.778377 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.778750 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.778833 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.783068 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:34775: BlockingRecv error: recv got EOF from 127.8.226.67:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260925 15:19:54.783373 18340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:19:55 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:55 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:55.250196 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:55.241834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:33509)
0925 15:19:55.242365 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:55.242381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:55.242489 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:55.242797 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:55.242800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:55.243151 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:55.243296 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:55.243303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.244221 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.244224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:55.245129 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:55.245139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.245798 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.245802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:55.245911 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:55.246445 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:55.246462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:55.249986 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":308,"threads_started":1}
W20260925 15:19:55.256325 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:55.241890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:33509 (local address 127.8.226.126:38373)
0925 15:19:55.242103 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:55.242107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:55.242452 (+ 345us) server_negotiation.cc:407] Connection header received
0925 15:19:55.242579 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:55.242582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:55.242661 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:55.242746 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:55.243421 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.244107 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.245257 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.245668 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.245760 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:55.250159 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:33509: BlockingRecv error: recv got EOF from 127.8.226.65:33509 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 25 15:19:55 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:55 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:55.604408 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:55.595496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:52019)
0925 15:19:55.596020 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:55.596035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:55.596186 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:55.596533 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:55.596538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:55.597033 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:55.597194 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:55.597202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.598227 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.598232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:55.599180 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:55.599190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.599908 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.599914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:55.600076 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:55.600648 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:55.600665 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:55.604069 (+ 3404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":308,"threads_started":1}
W20260925 15:19:55.604516 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:55.595559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:52019 (local address 127.8.226.126:38373)
0925 15:19:55.595792 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:55.595800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:55.596123 (+ 323us) server_negotiation.cc:407] Connection header received
0925 15:19:55.596324 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:55.596330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:55.596396 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:55.596501 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:55.597332 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.598084 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.599333 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.599733 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.599875 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:55.604278 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:52019: BlockingRecv error: recv got EOF from 127.8.226.66:52019 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:19:55 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:55 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:55.792567 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:55.783914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:57487)
0925 15:19:55.784649 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:55.784664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:55.784809 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:55.785219 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:55.785224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:55.785694 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:55.785859 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:55.785867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.786793 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.786797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:55.787742 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:55.787752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.788451 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.788455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:55.788560 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:55.789148 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:55.789163 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:55.792344 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":523,"threads_started":1}
W20260925 15:19:55.792726 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:55.784007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:57487 (local address 127.8.226.126:38373)
0925 15:19:55.784180 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:55.784187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:55.784751 (+ 564us) server_negotiation.cc:407] Connection header received
0925 15:19:55.784932 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:55.784937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:55.785049 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:55.785146 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:55.785979 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.786687 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.787856 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:55.788314 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:55.788400 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:55.792515 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:57487: BlockingRecv error: recv got EOF from 127.8.226.67:57487 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:19:56 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:56 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:56.260692 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.251115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:39061)
0925 15:19:56.251653 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.251671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.251813 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.252113 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.252118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.252577 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.252775 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.252786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.253777 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.253782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.255263 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.255276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.255877 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.255883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.256073 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.256694 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.256722 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.260484 (+ 3762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":310,"threads_started":1}
W20260925 15:19:56.260929 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:56.251141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:39061 (local address 127.8.226.126:38373)
0925 15:19:56.251376 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.251402 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.251763 (+ 361us) server_negotiation.cc:407] Connection header received
0925 15:19:56.251919 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.251923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.251985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.252064 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.252902 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.253659 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.255381 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.255750 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.255831 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.260668 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:39061: BlockingRecv error: recv got EOF from 127.8.226.65:39061 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:19:56 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:56 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:56.614323 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.605428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:40109)
0925 15:19:56.605966 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.605991 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.606096 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.606368 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.606370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.606731 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.606904 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.606910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.607868 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.607875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.609212 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.609229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.609913 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.609935 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.610056 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.610575 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.610592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.613983 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":300,"threads_started":1}
W20260925 15:19:56.614374 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:56.605502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:40109 (local address 127.8.226.126:38373)
0925 15:19:56.605675 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.605679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.606055 (+ 376us) server_negotiation.cc:407] Connection header received
0925 15:19:56.606193 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.606197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.606257 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.606333 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.607032 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.607731 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.609362 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.609788 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.609879 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.614160 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:40109: BlockingRecv error: recv got EOF from 127.8.226.66:40109 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 15:19:56 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:56 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:56.802147 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.793544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:60943)
0925 15:19:56.794216 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.794232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.794356 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.794740 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.794744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.795201 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.795467 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.795479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.796499 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.796503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.797417 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.797428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.798006 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.798010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.798127 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.798653 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.798668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.801837 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":334,"threads_started":1}
W20260925 15:19:56.802304 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:56.793907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:60943 (local address 127.8.226.126:38373)
0925 15:19:56.794087 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.794092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.794313 (+ 221us) server_negotiation.cc:407] Connection header received
0925 15:19:56.794472 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.794476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.794592 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.794691 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.795610 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.796390 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.797528 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.797890 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.797983 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.802068 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:60943: BlockingRecv error: recv got EOF from 127.8.226.67:60943 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 15:19:57 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:57 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:57.270895 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.261641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:55971)
0925 15:19:57.262102 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:57.262117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:57.262272 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:57.262613 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:57.262619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:57.263221 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:57.263427 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.263439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.264604 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.264608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.265606 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.265617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.266306 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.266310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.266426 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.266983 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:57.267000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:57.270676 (+ 3676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":246,"threads_started":1}
W20260925 15:19:57.271178 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.261751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:55971 (local address 127.8.226.126:38373)
0925 15:19:57.261942 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.261948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.262202 (+ 254us) server_negotiation.cc:407] Connection header received
0925 15:19:57.262400 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.262406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.262475 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.262576 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.263583 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.264468 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.265741 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.266186 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.266287 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.270922 (+ 4635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:55971: BlockingRecv error: recv got EOF from 127.8.226.65:55971 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 15:19:57 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:57 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:57.623971 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.615232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:59391)
0925 15:19:57.615718 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:57.615733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:57.615839 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:57.616131 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:57.616135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:57.616530 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:57.616691 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.616699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.617746 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.617749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.618624 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.618635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.619342 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.619345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.619463 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.619995 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:57.620011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:57.623584 (+ 3573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
W20260925 15:19:57.624040 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.615282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:59391 (local address 127.8.226.126:38373)
0925 15:19:57.615437 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.615442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.615800 (+ 358us) server_negotiation.cc:407] Connection header received
0925 15:19:57.615927 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.615931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.616007 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.616088 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.616817 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.617621 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.618770 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.619222 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.619322 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.623791 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:59391: BlockingRecv error: recv got EOF from 127.8.226.66:59391 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:19:57 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:57 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:57.811360 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.802948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:47455)
0925 15:19:57.803479 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:57.803494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:57.803608 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:57.803890 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:57.803893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:57.804236 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:57.804399 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.804407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.805479 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.805483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.806470 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.806481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.807071 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.807074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.807187 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.807811 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:57.807826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:57.811150 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":297,"threads_started":1}
W20260925 15:19:57.811607 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.803031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:47455 (local address 127.8.226.126:38373)
0925 15:19:57.803193 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.803197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.803561 (+ 364us) server_negotiation.cc:407] Connection header received
0925 15:19:57.803700 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.803705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.803774 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.803853 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.804515 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.805340 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.806583 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.806959 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.807044 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.811326 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:47455: BlockingRecv error: recv got EOF from 127.8.226.67:47455 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:19:58 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:58 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:58.280644 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.271890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:40499)
0925 15:19:58.272359 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:58.272374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:58.272525 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:58.272886 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:58.272891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:58.273308 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:58.273470 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:58.273478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.274509 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.274516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:58.275514 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:58.275524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.276241 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.276247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:58.276430 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.276973 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:58.276991 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:58.280303 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
W20260925 15:19:58.280742 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:58.271969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:40499 (local address 127.8.226.126:38373)
0925 15:19:58.272228 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.272234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.272468 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:19:58.272665 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.272670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.272741 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.272845 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:58.273611 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.274368 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.275651 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.276097 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.276238 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.280504 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:40499: BlockingRecv error: recv got EOF from 127.8.226.65:40499 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 15:19:58 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:58 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:58.633288 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.625073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:33549)
0925 15:19:58.625561 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:58.625575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:58.625672 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:58.625924 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:58.625927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:58.626272 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:58.626427 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:58.626434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.627357 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.627360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:58.628329 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:58.628339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.628913 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.628917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:58.629103 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.629707 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:58.629723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:58.633004 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
W20260925 15:19:58.633396 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:58.625134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:33549 (local address 127.8.226.126:38373)
0925 15:19:58.625354 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.625363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.625643 (+ 280us) server_negotiation.cc:407] Connection header received
0925 15:19:58.625754 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.625757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.625813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.625886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:58.626531 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.627251 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.628431 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.628803 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.628891 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.633181 (+ 4290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:33549: BlockingRecv error: recv got EOF from 127.8.226.66:33549 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 15:19:58 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:58 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:58.821060 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.812302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:33085)
0925 15:19:58.812803 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:58.812818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:58.813005 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:58.813396 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:58.813400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:58.813827 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:58.814042 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:58.814052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.815017 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.815024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:58.815986 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:58.815997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.816684 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.816689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:58.816865 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.817493 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:58.817509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:58.820757 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":264,"threads_started":1}
W20260925 15:19:58.821271 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:58.812350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:33085 (local address 127.8.226.126:38373)
0925 15:19:58.812569 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.812577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.812902 (+ 325us) server_negotiation.cc:407] Connection header received
0925 15:19:58.813128 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.813134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.813264 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.813371 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:58.814174 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.814887 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.816131 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.816544 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.816674 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.820930 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:33085: BlockingRecv error: recv got EOF from 127.8.226.67:33085 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:19:59 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:59 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:59.289924 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.281613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:52027)
0925 15:19:59.282280 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.282312 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.282415 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.282668 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.282670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.283036 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.283173 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.283180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.284188 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.284192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.285257 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.285273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.285847 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.285850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.285956 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.286439 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.286454 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.289618 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":458,"threads_started":1}
W20260925 15:19:59.290009 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.281697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:52027 (local address 127.8.226.126:38373)
0925 15:19:59.281920 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.281928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.282382 (+ 454us) server_negotiation.cc:407] Connection header received
0925 15:19:59.282494 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.282497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.282553 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.282629 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.283293 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.284060 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.285398 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.285731 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.285816 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.289804 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:52027: BlockingRecv error: recv got EOF from 127.8.226.65:52027 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:19:59 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:59 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:59.642457 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.634181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:51951)
0925 15:19:59.634842 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.634863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.635012 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.635355 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.635358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.635745 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.635882 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.635888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.636767 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.636770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.637706 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.637717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.638313 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.638316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.638428 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.638942 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.638958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.642127 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":349,"threads_started":1}
W20260925 15:19:59.642557 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.634381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:51951 (local address 127.8.226.126:38373)
0925 15:19:59.634530 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.634537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.634938 (+ 401us) server_negotiation.cc:407] Connection header received
0925 15:19:59.635122 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.635133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.635225 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.635311 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.635988 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.636660 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.637823 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.638204 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.638298 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.642327 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:51951: BlockingRecv error: recv got EOF from 127.8.226.66:51951 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 15:19:59 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:19:59 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:19:59.835420 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.822011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:42653)
0925 15:19:59.822563 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.822579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.822686 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.822969 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.822972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.823379 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.823518 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.823544 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.824389 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.824392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.825243 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.825253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.825802 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.825806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.825909 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.831614 (+ 5705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.831639 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.835104 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":322,"threads_started":1}
W20260925 15:19:59.835546 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.822059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:42653 (local address 127.8.226.126:38373)
0925 15:19:59.822220 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.822225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.822650 (+ 425us) server_negotiation.cc:407] Connection header received
0925 15:19:59.822778 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.822781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.822851 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.822929 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.823643 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.824284 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.825341 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.825696 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.825778 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.835317 (+ 9539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:42653: BlockingRecv error: recv got EOF from 127.8.226.67:42653 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:20:00 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:00 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:00.300009 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.290941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:44857)
0925 15:20:00.291441 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:00.291457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:00.291610 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:00.291948 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:00.291953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:00.292390 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:00.292537 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:00.292545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.293613 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.293620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:00.294560 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:00.294570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.295302 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.295308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:00.295488 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.296035 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:00.296052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:00.299775 (+ 3723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":265,"threads_started":1}
W20260925 15:20:00.300262 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.291012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:44857 (local address 127.8.226.126:38373)
0925 15:20:00.291268 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.291276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.291542 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:20:00.291731 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.291738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.291806 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.291918 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.292676 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.293476 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.294698 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.295143 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.295288 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.299990 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:44857: BlockingRecv error: recv got EOF from 127.8.226.65:44857 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:20:00 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:00 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:00.652136 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.643539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:39973)
0925 15:20:00.644031 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:00.644046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:00.644148 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:00.644419 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:00.644422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:00.644787 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:00.644992 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:00.645006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.645895 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.645898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:00.646754 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:00.646769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.647517 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.647521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:00.647766 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.648295 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:00.648310 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:00.651830 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":284,"threads_started":1}
W20260925 15:20:00.652187 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.643622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:39973 (local address 127.8.226.126:38373)
0925 15:20:00.643784 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.643788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.644126 (+ 338us) server_negotiation.cc:407] Connection header received
0925 15:20:00.644239 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.644249 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.644305 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.644381 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.645115 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.645778 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.646902 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.647335 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.647466 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.651979 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:39973: BlockingRecv error: recv got EOF from 127.8.226.66:39973 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:20:00 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:00 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:00.844264 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.836378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:41709)
0925 15:20:00.836779 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:00.836792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:00.836915 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:00.837208 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:00.837211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:00.837590 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:00.837746 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:00.837754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.838634 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.838637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:00.839521 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:00.839531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.840143 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.840147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:00.840266 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.840800 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:00.840815 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:00.844045 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":219,"threads_started":1}
W20260925 15:20:00.844540 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.836431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:41709 (local address 127.8.226.126:38373)
0925 15:20:00.836629 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.836635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.836868 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:20:00.837034 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.837037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.837094 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.837169 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.837850 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.838529 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.839633 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.840024 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.840108 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.844260 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:41709: BlockingRecv error: recv got EOF from 127.8.226.67:41709 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:20:01 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:01 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:01.310343 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:01.301128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:46263)
0925 15:20:01.301733 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:01.301748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:01.301893 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:01.302324 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:01.302329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:01.302828 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:01.303032 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:01.303041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.304135 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.304140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:01.305201 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:01.305211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.305999 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.306005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:01.306167 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:01.306836 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:01.306853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:01.310098 (+ 3245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":265,"threads_started":1}
W20260925 15:20:01.310470 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:01.301196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:46263 (local address 127.8.226.126:38373)
0925 15:20:01.301386 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:01.301391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:01.301959 (+ 568us) server_negotiation.cc:407] Connection header received
0925 15:20:01.302047 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:01.302054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:01.302147 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:01.302248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:01.303184 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.303984 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.305340 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.305857 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.305985 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:01.310289 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:46263: BlockingRecv error: recv got EOF from 127.8.226.65:46263 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 15:20:01 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:01 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:01.662377 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:01.653735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:41169)
0925 15:20:01.654354 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:01.654373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:01.654524 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:01.654907 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:01.654913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:01.655428 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:01.655583 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:01.655591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.656549 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.656552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:01.657543 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:01.657554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.658335 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.658339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:01.658447 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:01.658975 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:01.658991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:01.662146 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":317,"threads_started":1}
W20260925 15:20:01.662557 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:01.653781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:41169 (local address 127.8.226.126:38373)
0925 15:20:01.654010 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:01.654017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:01.654475 (+ 458us) server_negotiation.cc:407] Connection header received
0925 15:20:01.654639 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:01.654647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:01.654722 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:01.654842 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:01.655722 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.656415 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.657675 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.658212 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.658305 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:01.662334 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:41169: BlockingRecv error: recv got EOF from 127.8.226.66:41169 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:20:01 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:01 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:01.853549 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:01.845350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:48095)
0925 15:20:01.845892 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:01.845907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:01.846022 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:01.846314 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:01.846317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:01.846679 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:01.846844 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:01.846851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.847706 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.847709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:01.848643 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:01.848653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.849343 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.849347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:01.849450 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:01.850091 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:01.850108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:01.853270 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":261,"threads_started":1}
W20260925 15:20:01.853732 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:01.845389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:48095 (local address 127.8.226.126:38373)
0925 15:20:01.845549 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:01.845553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:01.845976 (+ 423us) server_negotiation.cc:407] Connection header received
0925 15:20:01.846102 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:01.846106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:01.846163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:01.846291 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:01.846957 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.847605 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.848756 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:01.849205 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:01.849292 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:01.853504 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:48095: BlockingRecv error: recv got EOF from 127.8.226.67:48095 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:20:02 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:02 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:02.319756 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:02.311342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:59237)
0925 15:20:02.311862 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:02.311879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:02.311995 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:02.312280 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:02.312283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:02.312772 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:02.312976 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:02.312985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.314105 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.314108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:02.314999 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:02.315008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.315596 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.315600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:02.315723 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:02.316353 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:02.316370 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:02.319511 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
W20260925 15:20:02.319934 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:02.311502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:59237 (local address 127.8.226.126:38373)
0925 15:20:02.311656 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:02.311660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:02.311945 (+ 285us) server_negotiation.cc:407] Connection header received
0925 15:20:02.312102 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:02.312105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:02.312161 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:02.312240 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:02.313107 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.313954 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.315099 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.315463 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.315547 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:02.319695 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:59237: BlockingRecv error: recv got EOF from 127.8.226.65:59237 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 15:20:02 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:02 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:02.672835 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:02.663446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:43977)
0925 15:20:02.664005 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:02.664019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:02.664171 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:02.664513 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:02.664517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:02.664912 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:02.665131 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:02.665139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.666313 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.666319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:02.667268 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:02.667279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.668003 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.668009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:02.668166 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:02.668907 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:02.668932 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:02.672474 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":335,"threads_started":1}
W20260925 15:20:02.672998 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:02.663558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:43977 (local address 127.8.226.126:38373)
0925 15:20:02.663823 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:02.663829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:02.664106 (+ 277us) server_negotiation.cc:407] Connection header received
0925 15:20:02.664282 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:02.664304 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:02.664384 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:02.664481 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:02.665262 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.666156 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.667417 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.667811 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.667949 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:02.672725 (+ 4776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:43977: BlockingRecv error: recv got EOF from 127.8.226.66:43977 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 15:20:02 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:02 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:02.863622 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:02.854431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:41353)
0925 15:20:02.854983 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:02.854998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:02.855147 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:02.855481 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:02.855485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:02.855901 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:02.856057 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:02.856065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.857097 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.857105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:02.858075 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:02.858086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.858818 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.858823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:02.858979 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:02.859607 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:02.859624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:02.863285 (+ 3661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":301,"threads_started":1}
W20260925 15:20:02.863700 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:02.854500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:41353 (local address 127.8.226.126:38373)
0925 15:20:02.854731 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:02.854738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:02.855077 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:20:02.855273 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:02.855280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:02.855352 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:02.855451 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:02.856198 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.856933 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.858215 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:02.858642 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:02.858789 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:02.863469 (+ 4680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:41353: BlockingRecv error: recv got EOF from 127.8.226.67:41353 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:20:03 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:03 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:03.329599 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:03.320759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:36359)
0925 15:20:03.321277 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:03.321292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:03.321439 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:03.321773 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:03.321777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:03.322180 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:03.322325 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:03.322333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.323353 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.323361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:03.324337 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:03.324348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.325051 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.325056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:03.325231 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:03.325760 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:03.325776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:03.329275 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":254,"threads_started":1}
W20260925 15:20:03.329751 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:03.320837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:36359 (local address 127.8.226.126:38373)
0925 15:20:03.321070 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:03.321077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:03.321378 (+ 301us) server_negotiation.cc:407] Connection header received
0925 15:20:03.321570 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:03.321575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:03.321643 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:03.321745 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:03.322477 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.323222 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.324477 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.324892 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.325038 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:03.329498 (+ 4460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:36359: BlockingRecv error: recv got EOF from 127.8.226.65:36359 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 15:20:03 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:03 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:03.682994 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:03.673790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:52491)
0925 15:20:03.674239 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:03.674262 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:03.674417 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:03.674754 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:03.674759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:03.675334 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:03.675485 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:03.675493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.676491 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.676497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:03.677513 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:03.677524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.678232 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.678237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:03.678444 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:03.679080 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:03.679096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:03.682764 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
W20260925 15:20:03.683180 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:03.673806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:52491 (local address 127.8.226.126:38373)
0925 15:20:03.673996 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:03.674004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:03.674343 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:20:03.674546 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:03.674552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:03.674622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:03.674723 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:03.675625 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.676362 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.677670 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.678084 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.678218 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:03.682931 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:52491: BlockingRecv error: recv got EOF from 127.8.226.66:52491 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:20:03 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:03 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:03.873360 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:03.864547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:42263)
0925 15:20:03.865096 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:03.865121 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:03.865268 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:03.865705 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:03.865711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:03.866229 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:03.866438 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:03.866451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.867512 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.867516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:03.868455 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:03.868466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.869157 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.869161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:03.869274 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:03.869836 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:03.869853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:03.873119 (+ 3266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":253,"threads_started":1}
W20260925 15:20:03.873505 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:03.864585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:42263 (local address 127.8.226.126:38373)
0925 15:20:03.864822 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:03.864830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:03.865199 (+ 369us) server_negotiation.cc:407] Connection header received
0925 15:20:03.865399 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:03.865403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:03.865500 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:03.865628 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:03.866577 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.867387 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.868577 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:03.869016 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:03.869110 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:03.873285 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:42263: BlockingRecv error: recv got EOF from 127.8.226.67:42263 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:20:04 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:04 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:04.339300 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:04.330693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:48859)
0925 15:20:04.331142 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:04.331158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:04.331266 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:04.331544 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:04.331547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:04.331893 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:04.332040 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:04.332048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.333224 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.333237 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:04.334469 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:04.334481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.335101 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.335105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:04.335209 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:04.335733 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:04.335755 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:04.338980 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":239,"threads_started":1}
W20260925 15:20:04.339422 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:04.330785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:48859 (local address 127.8.226.126:38373)
0925 15:20:04.331026 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:04.331031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:04.331241 (+ 210us) server_negotiation.cc:407] Connection header received
0925 15:20:04.331361 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:04.331365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:04.331425 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:04.331503 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:04.332159 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.332877 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.334586 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.334983 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.335071 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:04.339165 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:48859: BlockingRecv error: recv got EOF from 127.8.226.65:48859 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 15:20:04 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:04 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:04.693652 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:04.683882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:59195)
0925 15:20:04.684369 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:04.684385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:04.684536 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:04.684852 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:04.684856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:04.685236 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:04.685394 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:04.685402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.686404 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.686412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:04.687372 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:04.687382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.688033 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.688039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:04.688200 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:04.688911 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:04.688939 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:04.693401 (+ 4462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":247,"threads_started":1}
W20260925 15:20:04.694159 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:04.683948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:59195 (local address 127.8.226.126:38373)
0925 15:20:04.684188 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:04.684195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:04.684474 (+ 279us) server_negotiation.cc:407] Connection header received
0925 15:20:04.684654 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:04.684662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:04.684722 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:04.684820 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:04.685545 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.686273 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.687509 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.687889 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.688019 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:04.693581 (+ 5562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:59195: BlockingRecv error: recv got EOF from 127.8.226.66:59195 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 25 15:20:04 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:04 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:04.889074 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:04.874290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:57211)
0925 15:20:04.874823 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:04.874838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:04.874946 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:04.875189 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:04.875192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:04.875525 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:04.875678 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:04.875686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.876890 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.876893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:04.878032 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:04.878043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.878742 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.878747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:04.878912 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:04.884890 (+ 5978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:04.884909 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:04.888735 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
W20260925 15:20:04.889257 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:04.874396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:57211 (local address 127.8.226.126:38373)
0925 15:20:04.874638 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:04.874642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:04.874906 (+ 264us) server_negotiation.cc:407] Connection header received
0925 15:20:04.875035 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:04.875039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:04.875097 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:04.875175 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:04.875817 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.876709 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.878164 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:04.878600 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:04.878729 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:04.888988 (+ 10259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:57211: BlockingRecv error: recv got EOF from 127.8.226.67:57211 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 25 15:20:05 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:05 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:05.348730 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:05.340277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:57661)
0925 15:20:05.340782 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:05.340798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:05.340904 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:05.341190 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:05.341193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:05.341573 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:05.341732 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:05.341739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.342676 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.342680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:05.343597 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:05.343607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.344248 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.344252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:05.344357 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:05.344921 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:05.344936 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:05.348516 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":279,"threads_started":1}
W20260925 15:20:05.348891 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:05.340329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:57661 (local address 127.8.226.126:38373)
0925 15:20:05.340494 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:05.340498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:05.340866 (+ 368us) server_negotiation.cc:407] Connection header received
0925 15:20:05.341013 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:05.341017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:05.341072 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:05.341150 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:05.341853 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.342575 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.343710 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.344127 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.344216 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:05.348691 (+ 4475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:57661: BlockingRecv error: recv got EOF from 127.8.226.65:57661 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 15:20:05 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:05 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:05.703938 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:05.694612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:46149)
0925 15:20:05.695654 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:05.695672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:05.695872 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:05.696261 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:05.696266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:05.696715 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:05.696884 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:05.696894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.697948 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.697952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:05.698949 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:05.698960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.699592 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.699596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:05.699713 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:05.700334 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:05.700351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:05.703711 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":815,"threads_started":1}
W20260925 15:20:05.704115 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:05.694671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:46149 (local address 127.8.226.126:38373)
0925 15:20:05.694837 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:05.694841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:05.695759 (+ 918us) server_negotiation.cc:407] Connection header received
0925 15:20:05.696015 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:05.696020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:05.696097 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:05.696188 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:05.697069 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.697837 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.699062 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.699473 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.699560 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:05.703876 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:46149: BlockingRecv error: recv got EOF from 127.8.226.66:46149 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:20:05 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:05 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:05.898494 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:05.890188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:44087)
0925 15:20:05.890730 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:05.890746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:05.890862 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:05.891134 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:05.891137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:05.891543 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:05.891725 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:05.891734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.892604 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.892607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:05.893644 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:05.893655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.894499 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.894505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:05.894664 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:05.895195 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:05.895213 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:05.898301 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
W20260925 15:20:05.898672 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:05.890253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:44087 (local address 127.8.226.126:38373)
0925 15:20:05.890406 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:05.890411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:05.890816 (+ 405us) server_negotiation.cc:407] Connection header received
0925 15:20:05.890950 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:05.890954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:05.891012 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:05.891090 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:05.891838 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.892503 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.893804 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:05.894355 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:05.894480 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:05.898450 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:44087: BlockingRecv error: recv got EOF from 127.8.226.67:44087 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:20:06 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:06 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:06.357993 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:06.349615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:43009)
0925 15:20:06.350085 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:06.350101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:06.350224 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:06.350505 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:06.350509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:06.350898 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:06.351060 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:06.351068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.352007 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.352011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:06.352927 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:06.352937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.353580 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.353583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:06.353709 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:06.354273 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:06.354290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:06.357575 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
W20260925 15:20:06.358225 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:06.349650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:43009 (local address 127.8.226.126:38373)
0925 15:20:06.349890 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:06.349899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:06.350184 (+ 285us) server_negotiation.cc:407] Connection header received
0925 15:20:06.350308 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:06.350311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:06.350381 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:06.350460 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:06.351174 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.351894 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.353057 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.353466 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.353553 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:06.357856 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:43009: BlockingRecv error: recv got EOF from 127.8.226.65:43009 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 15:20:06 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:06 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:06.713853 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:06.704893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:48203)
0925 15:20:06.705389 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:06.705403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:06.705545 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:06.705947 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:06.705951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:06.706353 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:06.706520 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:06.706528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.707902 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.707909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:06.708779 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:06.708790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.709441 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.709445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:06.709583 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:06.710130 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:06.710146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:06.713554 (+ 3408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
W20260925 15:20:06.714040 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:06.704937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:48203 (local address 127.8.226.126:38373)
0925 15:20:06.705160 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:06.705168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:06.705479 (+ 311us) server_negotiation.cc:407] Connection header received
0925 15:20:06.705718 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:06.705723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:06.705818 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:06.705914 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:06.706658 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.707762 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.708888 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.709336 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.709417 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:06.713792 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:48203: BlockingRecv error: recv got EOF from 127.8.226.66:48203 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 15:20:06 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:06 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:06.908228 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:06.899363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:45955)
0925 15:20:06.899886 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:06.899901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:06.900056 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:06.900392 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:06.900398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:06.900820 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:06.901008 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:06.901017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.902031 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.902036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:06.902921 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:06.902931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.903658 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.903664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:06.903844 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:06.904458 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:06.904481 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:06.908001 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":259,"threads_started":1}
W20260925 15:20:06.908390 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:06.899434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:45955 (local address 127.8.226.126:38373)
0925 15:20:06.899654 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:06.899662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:06.899990 (+ 328us) server_negotiation.cc:407] Connection header received
0925 15:20:06.900176 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:06.900181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:06.900249 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:06.900348 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:06.901152 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.901917 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.903077 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:06.903483 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:06.903607 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:06.908179 (+ 4572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:45955: BlockingRecv error: recv got EOF from 127.8.226.67:45955 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 15:20:07 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:07 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:07.367376 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:07.359007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:43147)
0925 15:20:07.359537 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:07.359552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:07.359659 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:07.359934 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:07.359937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:07.360315 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:07.360467 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:07.360475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.361497 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.361501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:07.362383 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:07.362393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.363000 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.363003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:07.363127 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:07.363687 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:07.363711 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:07.367149 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":312,"threads_started":1}
W20260925 15:20:07.367581 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:07.359063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:43147 (local address 127.8.226.126:38373)
0925 15:20:07.359358 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:07.359363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:07.359621 (+ 258us) server_negotiation.cc:407] Connection header received
0925 15:20:07.359750 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:07.359754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:07.359812 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:07.359889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:07.360584 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.361382 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.362486 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.362885 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.362972 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:07.367334 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:43147: BlockingRecv error: recv got EOF from 127.8.226.65:43147 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 25 15:20:07 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:07 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.66@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:07.723222 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:07.714937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.66:55657)
0925 15:20:07.715417 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:07.715431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:07.715529 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:07.715942 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:07.715947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:07.716479 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:07.716718 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:07.716734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.717739 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.717743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:07.718556 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:07.718566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.719137 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.719141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:07.719254 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:07.719799 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:07.719816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:07.722884 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
W20260925 15:20:07.723321 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:07.715039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:55657 (local address 127.8.226.126:38373)
0925 15:20:07.715205 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:07.715209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:07.715508 (+ 299us) server_negotiation.cc:407] Connection header received
0925 15:20:07.715661 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:07.715664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:07.715747 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:07.715867 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:07.716863 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.717625 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.718667 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.719035 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.719114 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:07.723061 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.66:55657: BlockingRecv error: recv got EOF from 127.8.226.66:55657 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 15:20:07 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:07 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.67@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:07.918053 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:07.909321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.67:59825)
0925 15:20:07.909783 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:07.909799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:07.909946 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:07.910311 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:07.910316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:07.910739 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:07.910874 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:07.910882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.912065 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.912070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:07.913450 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:07.913462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.914102 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.914108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:07.914291 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:07.914811 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:07.914826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:07.917843 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":272,"threads_started":1}
W20260925 15:20:07.918188 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:07.909378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:59825 (local address 127.8.226.126:38373)
0925 15:20:07.909598 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:07.909624 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:07.909883 (+ 259us) server_negotiation.cc:407] Connection header received
0925 15:20:07.910079 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:07.910101 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:07.910168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:07.910275 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:07.911014 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.911905 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.913579 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:07.913966 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:07.914084 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:07.918011 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.67:59825: BlockingRecv error: recv got EOF from 127.8.226.67:59825 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 15:20:08 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:08 dist-test-slave-8vw7 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.226.65@KRBTEST.COM for oryx/127.8.226.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:08.377306 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:08.368560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38373 (local address 127.8.226.65:38845)
0925 15:20:08.369115 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:08.369134 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:08.369271 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:08.369646 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:08.369652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:08.370064 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:08.370230 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:08.370238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.371245 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.371251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:08.372195 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:08.372206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.372947 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.372976 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:08.373166 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:08.373812 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:08.373829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:08.377053 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.226.126:38373: Server oryx/127.8.226.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
W20260925 15:20:08.377424 18343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:08.368618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:38845 (local address 127.8.226.126:38373)
0925 15:20:08.368886 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:08.368893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:08.369249 (+ 356us) server_negotiation.cc:407] Connection header received
0925 15:20:08.369425 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:08.369430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:08.369503 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:08.369617 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:08.370374 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.371084 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.372335 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.372804 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.372937 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:08.377191 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.226.65:38845: BlockingRecv error: recv got EOF from 127.8.226.65:38845 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 15:20:08.673426 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17927
I20260925 15:20:08.685277 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:08.674527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.8.226.65:35013)
0925 15:20:08.675083 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:08.675089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:08.675113 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:20:08.675178 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:08.675183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:08.675405 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:08.675551 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:08.676841 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.677873 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.678908 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.679416 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.679506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:08.682132 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:08.682140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:08.682162 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:08.682200 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:08.684688 (+ 2488us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.226.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:20:08.685008 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49044: Request ticket server kudu/127.8.226.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":288,"threads_started":1}
W20260925 15:20:08.685376 18389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49044: Request ticket server kudu/127.8.226.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:20:08.685456 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17927: Not authorized: Client connection negotiation failed: client connection to 127.8.226.65:35013: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.226.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:20:08.685637 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17927
I20260925 15:20:08.701803 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18078
I20260925 15:20:08.713984 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:08.702709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.8.226.66:33375)
0925 15:20:08.703376 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:08.703385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:08.703417 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:20:08.703496 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:08.703505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:08.703844 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:08.704133 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:08.706038 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.707015 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.708158 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.708658 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.708790 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:08.711073 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:08.711081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:08.711092 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:08.711130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:08.713297 (+ 2167us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.226.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:20:08.713617 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58118: Request ticket server kudu/127.8.226.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":310,"threads_started":1}
W20260925 15:20:08.714015 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18078: Not authorized: Client connection negotiation failed: client connection to 127.8.226.66:33375: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.226.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:20:08.714094 18390 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58118: Request ticket server kudu/127.8.226.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:20:08.714160 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18078
I20260925 15:20:08.730072 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18211
I20260925 15:20:08.742002 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:08.731073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.8.226.67:44995)
0925 15:20:08.731607 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:08.731614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:08.731636 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:20:08.731698 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:08.731704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:08.731925 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:08.732144 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:08.733575 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.734576 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.735777 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:08.736297 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:08.736441 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:08.738955 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:08.738963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:08.738975 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:08.739009 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:08.741325 (+ 2316us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.226.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:20:08.741653 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37508: Request ticket server kudu/127.8.226.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
W20260925 15:20:08.741972 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18211: Not authorized: Client connection negotiation failed: client connection to 127.8.226.67:44995: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.226.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:20:08.742086 18391 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37508: Request ticket server kudu/127.8.226.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:20:08.742121 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18211
I20260925 15:20:08.757337 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 17858
I20260925 15:20:08.758493 17920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:08.871567 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 17858
2026-09-25T15:20:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20751 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:08Z Disabled control of system clock
I20260925 15:20:08.941896 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:46659
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:46659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:20:09.249804 18396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:09.250058 18396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:09.250201 18396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:09.256980 18396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:09.257086 18396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:09.257155 18396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:09.257236 18396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:09.257316 18396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:09.265888 18396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:46659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:46659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:09.267060 18396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:09.268730 18396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:09.280072 18401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:09.280160 18404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:09.280162 18402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:09.280777 18396 server_base.cc:1034] running on GCE node
I20260925 15:20:09.281481 18396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:09.282627 18396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:09.283847 18396 hybrid_clock.cc:648] HybridClock initialized: now 1790349609283823 us; error 48 us; skew 500 ppm
I20260925 15:20:09.284168 18396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:09.286258 18396 webserver.cc:460] Webserver started at http://127.8.226.126:46649/ using document root <none> and password file <none>
I20260925 15:20:09.286886 18396 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:09.287009 18396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:09.287312 18396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:09.290189 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "6a81ceb5b3614eb4acb5d06137592e69"
format_stamp: "Formatted at 2026-09-25 15:20:09 on dist-test-slave-8vw7"
server_key: "b9a7cbfcdb5726e672f4ecf9d1a5267f"
server_key_iv: "9a44ac36826dafccc3bf05b2509fed48"
server_key_version: "encryptionkey@0"
I20260925 15:20:09.290928 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "6a81ceb5b3614eb4acb5d06137592e69"
format_stamp: "Formatted at 2026-09-25 15:20:09 on dist-test-slave-8vw7"
server_key: "b9a7cbfcdb5726e672f4ecf9d1a5267f"
server_key_iv: "9a44ac36826dafccc3bf05b2509fed48"
server_key_version: "encryptionkey@0"
I20260925 15:20:09.295644 18396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:20:09.298888 18410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:09.300103 18396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:09.300271 18396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "6a81ceb5b3614eb4acb5d06137592e69"
format_stamp: "Formatted at 2026-09-25 15:20:09 on dist-test-slave-8vw7"
server_key: "b9a7cbfcdb5726e672f4ecf9d1a5267f"
server_key_iv: "9a44ac36826dafccc3bf05b2509fed48"
server_key_version: "encryptionkey@0"
I20260925 15:20:09.300469 18396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:09.320132 18396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:09.320935 18396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:09.321241 18396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:09.338526 18396 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:46659
I20260925 15:20:09.338550 18461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:46659 every 8 connection(s)
I20260925 15:20:09.339740 18396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:09.343724 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:09.350131 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18396
I20260925 15:20:09.350327 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:20:09.351054 9097 external_mini_cluster.cc:1428] Setting key 938de1d6f17d0ccc58dec6d3fb8f0c55
I20260925 15:20:09.354194 18462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69: Bootstrap starting.
I20260925 15:20:09.357191 18462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:09.358204 18462 log.cc:824] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:09.360786 18462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69: No bootstrap required, opened a new log
I20260925 15:20:09.363919 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:09.353065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.8.226.126:46659)
0925 15:20:09.353990 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:09.354001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:09.354036 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:20:09.355739 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:09.355761 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:09.356196 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:09.356549 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:09.357103 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:09.358454 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:09.359755 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:09.360415 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:09.360602 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:09.361801 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:09.361808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:09.361820 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:09.362343 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:09.362415 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:09.362434 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:09.362551 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:09.362781 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:20:09.363038 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":328,"threads_started":1}
I20260925 15:20:09.368557 18462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a81ceb5b3614eb4acb5d06137592e69" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46659 } }
I20260925 15:20:09.368896 18462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:09.369009 18462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a81ceb5b3614eb4acb5d06137592e69, State: Initialized, Role: FOLLOWER
I20260925 15:20:09.369499 18462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a81ceb5b3614eb4acb5d06137592e69" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46659 } }
I20260925 15:20:09.369774 18462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:09.369901 18462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:09.370072 18462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:09.372467 18462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a81ceb5b3614eb4acb5d06137592e69" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46659 } }
I20260925 15:20:09.373008 18462 leader_election.cc:302] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a81ceb5b3614eb4acb5d06137592e69; no voters:
I20260925 15:20:09.373545 18462 leader_election.cc:288] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:09.373708 18467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:09.374446 18467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [term 1 LEADER]: Becoming Leader. State: Replica: 6a81ceb5b3614eb4acb5d06137592e69, State: Running, Role: LEADER
I20260925 15:20:09.374933 18467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a81ceb5b3614eb4acb5d06137592e69" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46659 } }
I20260925 15:20:09.375483 18462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:09.377950 18468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a81ceb5b3614eb4acb5d06137592e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a81ceb5b3614eb4acb5d06137592e69" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46659 } } }
I20260925 15:20:09.377928 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a81ceb5b3614eb4acb5d06137592e69. Latest consensus state: current_term: 1 leader_uuid: "6a81ceb5b3614eb4acb5d06137592e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a81ceb5b3614eb4acb5d06137592e69" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46659 } } }
I20260925 15:20:09.378329 18468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:09.378386 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:09.379680 18475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:09.385094 18475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:09.392699 18475 catalog_manager.cc:1347] Generated new cluster ID: 268db9f1e33f42289eabecb8c4cfb50c
I20260925 15:20:09.392800 18475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:09.413702 18475 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:09.415058 18475 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:09.421429 18475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a81ceb5b3614eb4acb5d06137592e69: Generated new TSK 0
I20260925 15:20:09.422194 18475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:09.431596 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46659
--builtin_ntp_servers=127.8.226.84:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:20:09.741526 18486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:09.741873 18486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:09.742028 18486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:09.748844 18486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:09.748930 18486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:09.749112 18486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:20:09.758121 18486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:09.759626 18486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:09.761540 18486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:09.773950 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:09.774019 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:09.773998 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:09.774729 18486 server_base.cc:1034] running on GCE node
I20260925 15:20:09.775358 18486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:09.776264 18486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:09.777479 18486 hybrid_clock.cc:648] HybridClock initialized: now 1790349609777417 us; error 64 us; skew 500 ppm
I20260925 15:20:09.777788 18486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:09.779809 18486 webserver.cc:460] Webserver started at http://127.8.226.65:38353/ using document root <none> and password file <none>
I20260925 15:20:09.780478 18486 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:09.780591 18486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:09.780905 18486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:09.783581 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "aa89fc0d84354fa5970dfb31b6fd5f60"
format_stamp: "Formatted at 2026-09-25 15:20:09 on dist-test-slave-8vw7"
server_key: "387416afd28ab4b876dc796efcc193c2"
server_key_iv: "e1de4547026fe12449a73b5aa8df3f84"
server_key_version: "encryptionkey@0"
I20260925 15:20:09.784315 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "aa89fc0d84354fa5970dfb31b6fd5f60"
format_stamp: "Formatted at 2026-09-25 15:20:09 on dist-test-slave-8vw7"
server_key: "387416afd28ab4b876dc796efcc193c2"
server_key_iv: "e1de4547026fe12449a73b5aa8df3f84"
server_key_version: "encryptionkey@0"
I20260925 15:20:09.788784 18486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:20:09.791975 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:09.793311 18486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:09.793476 18486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "aa89fc0d84354fa5970dfb31b6fd5f60"
format_stamp: "Formatted at 2026-09-25 15:20:09 on dist-test-slave-8vw7"
server_key: "387416afd28ab4b876dc796efcc193c2"
server_key_iv: "e1de4547026fe12449a73b5aa8df3f84"
server_key_version: "encryptionkey@0"
I20260925 15:20:09.793679 18486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:09.824098 18486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:09.824941 18486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:09.825253 18486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:09.826151 18486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:09.827491 18486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:09.827605 18486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:09.827734 18486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:09.827824 18486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:09.859158 18486 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43765
I20260925 15:20:09.859239 18612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43765 every 8 connection(s)
I20260925 15:20:09.860441 18486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:09.863973 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18486
I20260925 15:20:09.864187 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:09.864923 9097 external_mini_cluster.cc:1428] Setting key 125e3c85f8a09e925cf65344d6ebb9e8
I20260925 15:20:09.867655 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46659
--builtin_ntp_servers=127.8.226.84:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:20:09.873695 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:09.863030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:40629 (local address 127.8.226.126:46659)
0925 15:20:09.863255 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:09.863262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:09.864788 (+ 1526us) server_negotiation.cc:407] Connection header received
0925 15:20:09.866800 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:09.866806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:09.866881 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:09.867004 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:09.868033 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:09.869248 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:09.870706 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:09.871399 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:09.871531 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:09.872747 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:09.872755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:09.872763 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:09.872787 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:09.872830 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:09.872847 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:09.872935 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:09.873192 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:20:09.873305 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:20:09.874457 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:09.863592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:46659 (local address 127.8.226.65:40629)
0925 15:20:09.864600 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:09.864660 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:09.866495 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:09.867184 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:09.867192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:09.867276 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:09.867871 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:09.867896 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:09.869383 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:09.869387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:09.870567 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:09.870578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:09.871578 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:09.871582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:09.871771 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:09.872454 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:09.872457 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:09.872495 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:09.872564 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:09.872939 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:09.872946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:09.872990 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:20:09.873143 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 15:20:09.873417 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
I20260925 15:20:09.875919 18613 heartbeater.cc:347] Connected to a master server at 127.8.226.126:46659
I20260925 15:20:09.876211 18613 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:09.876926 18613 heartbeater.cc:511] Master 127.8.226.126:46659 requested a full tablet report, sending...
I20260925 15:20:09.878878 18427 ts_manager.cc:194] Registered new tserver with Master: aa89fc0d84354fa5970dfb31b6fd5f60 (127.8.226.65:43765)
I20260925 15:20:09.880607 18427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.65:40629
W20260925 15:20:10.180044 18616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:10.180387 18616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:10.180533 18616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:10.187498 18616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:10.187628 18616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:10.187811 18616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:20:10.197106 18616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:10.198468 18616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:10.200212 18616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:10.212810 18621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:10.212893 18624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:10.212913 18622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:10.218245 18616 server_base.cc:1034] running on GCE node
I20260925 15:20:10.219014 18616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:10.219921 18616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:10.221130 18616 hybrid_clock.cc:648] HybridClock initialized: now 1790349610221087 us; error 63 us; skew 500 ppm
I20260925 15:20:10.221436 18616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:10.229246 18616 webserver.cc:460] Webserver started at http://127.8.226.66:32807/ using document root <none> and password file <none>
I20260925 15:20:10.229939 18616 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:10.230063 18616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:10.230381 18616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:10.233129 18616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "0299f255f082498eb670dcbf4a93d2c9"
format_stamp: "Formatted at 2026-09-25 15:20:10 on dist-test-slave-8vw7"
server_key: "63d778372ae8555485f0bd571c1facab"
server_key_iv: "85dc0d84007004788fadd9c73a73abda"
server_key_version: "encryptionkey@0"
I20260925 15:20:10.233922 18616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "0299f255f082498eb670dcbf4a93d2c9"
format_stamp: "Formatted at 2026-09-25 15:20:10 on dist-test-slave-8vw7"
server_key: "63d778372ae8555485f0bd571c1facab"
server_key_iv: "85dc0d84007004788fadd9c73a73abda"
server_key_version: "encryptionkey@0"
I20260925 15:20:10.238809 18616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260925 15:20:10.241869 18630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:10.243036 18616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:20:10.243212 18616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "0299f255f082498eb670dcbf4a93d2c9"
format_stamp: "Formatted at 2026-09-25 15:20:10 on dist-test-slave-8vw7"
server_key: "63d778372ae8555485f0bd571c1facab"
server_key_iv: "85dc0d84007004788fadd9c73a73abda"
server_key_version: "encryptionkey@0"
I20260925 15:20:10.243445 18616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:10.257130 18616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:10.257866 18616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:10.258133 18616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:10.258975 18616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:10.260373 18616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:10.260456 18616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:10.260530 18616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:10.260583 18616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:10.292559 18616 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:46097
I20260925 15:20:10.292606 18742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:46097 every 8 connection(s)
I20260925 15:20:10.294028 18616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:10.299908 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18616
I20260925 15:20:10.300072 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:10.300498 9097 external_mini_cluster.cc:1428] Setting key 49fd521d00c27f7eafda977d36358681
I20260925 15:20:10.303023 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46659
--builtin_ntp_servers=127.8.226.84:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:20:10.306316 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.296474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:54893 (local address 127.8.226.126:46659)
0925 15:20:10.296644 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:10.296648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:10.298188 (+ 1540us) server_negotiation.cc:407] Connection header received
0925 15:20:10.299911 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:10.299917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:10.300008 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:10.300128 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:10.301311 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.302032 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.303595 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.304062 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.304196 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.305563 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:10.305568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:10.305573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:10.305591 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:10.305628 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:10.305639 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:10.305725 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:10.305935 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:20:10.306020 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:20:10.307265 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.297147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:46659 (local address 127.8.226.66:54893)
0925 15:20:10.298029 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:10.298067 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:10.299648 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:10.300348 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:10.300356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:10.300439 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:10.301134 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:10.301161 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.302187 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.302192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:10.303456 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:10.303467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.304284 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.304291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:10.304501 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.305290 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:10.305293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:10.305326 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:10.305393 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:10.305735 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:10.305747 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:10.305754 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:20:10.305909 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 15:20:10.306243 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":250,"threads_started":1}
I20260925 15:20:10.308480 18743 heartbeater.cc:347] Connected to a master server at 127.8.226.126:46659
I20260925 15:20:10.308748 18743 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:10.309476 18743 heartbeater.cc:511] Master 127.8.226.126:46659 requested a full tablet report, sending...
I20260925 15:20:10.310940 18427 ts_manager.cc:194] Registered new tserver with Master: 0299f255f082498eb670dcbf4a93d2c9 (127.8.226.66:46097)
I20260925 15:20:10.312126 18427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.66:54893
W20260925 15:20:10.611764 18746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:10.611982 18746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:10.612057 18746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:10.618568 18746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:10.618643 18746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:10.618754 18746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:20:10.626569 18746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:10.627561 18746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:10.629076 18746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:10.641048 18752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:10.640931 18751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:10.641058 18754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:10.642035 18746 server_base.cc:1034] running on GCE node
I20260925 15:20:10.642513 18746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:10.643323 18746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:10.644501 18746 hybrid_clock.cc:648] HybridClock initialized: now 1790349610644446 us; error 56 us; skew 500 ppm
I20260925 15:20:10.644744 18746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:10.646900 18746 webserver.cc:460] Webserver started at http://127.8.226.67:43059/ using document root <none> and password file <none>
I20260925 15:20:10.647560 18746 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:10.647642 18746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:10.647938 18746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:10.650501 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "a63d46e8fbcb464aaccbf9db529009a1"
format_stamp: "Formatted at 2026-09-25 15:20:10 on dist-test-slave-8vw7"
server_key: "f54d59f8a99e4d44fa2c5c374a75c66c"
server_key_iv: "7a960eabcf93401262e6c22d20c614fc"
server_key_version: "encryptionkey@0"
I20260925 15:20:10.651180 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "a63d46e8fbcb464aaccbf9db529009a1"
format_stamp: "Formatted at 2026-09-25 15:20:10 on dist-test-slave-8vw7"
server_key: "f54d59f8a99e4d44fa2c5c374a75c66c"
server_key_iv: "7a960eabcf93401262e6c22d20c614fc"
server_key_version: "encryptionkey@0"
I20260925 15:20:10.655836 18746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:20:10.658845 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:10.659991 18746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:10.660112 18746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "a63d46e8fbcb464aaccbf9db529009a1"
format_stamp: "Formatted at 2026-09-25 15:20:10 on dist-test-slave-8vw7"
server_key: "f54d59f8a99e4d44fa2c5c374a75c66c"
server_key_iv: "7a960eabcf93401262e6c22d20c614fc"
server_key_version: "encryptionkey@0"
I20260925 15:20:10.660267 18746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:10.687541 18746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:10.688273 18746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:10.688499 18746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:10.689355 18746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:10.690654 18746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:10.690742 18746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:10.690821 18746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:10.690872 18746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:10.722038 18746 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:44485
I20260925 15:20:10.722117 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:44485 every 8 connection(s)
I20260925 15:20:10.723232 18746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:10.725204 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18746
I20260925 15:20:10.725364 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:10.725927 9097 external_mini_cluster.cc:1428] Setting key df6773d283b4676ed006761d605fec46
I20260925 15:20:10.734974 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.725769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:47503 (local address 127.8.226.126:46659)
0925 15:20:10.725984 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:10.725992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:10.727571 (+ 1579us) server_negotiation.cc:407] Connection header received
0925 15:20:10.729564 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:10.729568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:10.729637 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:10.729745 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:10.730692 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.731404 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.732695 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.733092 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.733175 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.734367 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:10.734371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:10.734374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:10.734389 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:10.734415 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:10.734425 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:10.734497 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:10.734687 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 15:20:10.734759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:20:10.735947 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.726319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:46659 (local address 127.8.226.67:47503)
0925 15:20:10.727409 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:10.727466 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:10.729338 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:10.729928 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:10.729935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:10.730011 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:10.730561 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:10.730577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.731512 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.731516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:10.732592 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:10.732601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.733246 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.733250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:10.733451 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.734125 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:10.734127 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:10.734161 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:10.734226 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:10.734511 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:10.734518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:10.734524 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:10.734664 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:20:10.734922 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":335,"threads_started":1}
I20260925 15:20:10.737146 18873 heartbeater.cc:347] Connected to a master server at 127.8.226.126:46659
I20260925 15:20:10.737426 18873 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:10.738075 18873 heartbeater.cc:511] Master 127.8.226.126:46659 requested a full tablet report, sending...
I20260925 15:20:10.739378 18427 ts_manager.cc:194] Registered new tserver with Master: a63d46e8fbcb464aaccbf9db529009a1 (127.8.226.67:44485)
I20260925 15:20:10.740319 18427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.67:47503
I20260925 15:20:10.740762 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:20:10.746645 18465 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:10.745101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.8.226.126:46659)
0925 15:20:10.745250 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:10.745254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:10.745744 (+ 490us) server_negotiation.cc:407] Connection header received
0925 15:20:10.745888 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:10.745892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:10.745950 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:10.746063 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:10.746359 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51572: BlockingRecv error: recv got EOF from 127.0.0.1:51572 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:20:10.746991 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18486
I20260925 15:20:10.754297 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.747848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.8.226.65:43765)
0925 15:20:10.748414 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:10.748420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:10.748441 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:20:10.748495 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:10.748500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:10.748659 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:10.748821 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:10.749194 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.750153 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.751324 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.751827 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.751955 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.752918 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:10.752924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:10.752934 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:10.753518 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:10.753568 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:10.753586 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:10.753706 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:10.753855 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:20:10.753975 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
I20260925 15:20:10.755321 18608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:10.833756 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18486
I20260925 15:20:10.850093 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18616
I20260925 15:20:10.856870 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.850936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.8.226.66:46097)
0925 15:20:10.851455 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:10.851461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:10.851486 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:20:10.851560 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:10.851566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:10.851784 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:10.851908 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:10.852229 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.853182 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.854300 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.854811 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.854894 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.855725 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:10.855733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:10.855745 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:10.856285 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:10.856327 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:10.856344 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:10.856428 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:10.856510 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 15:20:10.856605 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260925 15:20:10.858068 18738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:10.935300 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18616
I20260925 15:20:10.951105 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18746
I20260925 15:20:10.958599 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:10.952029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.8.226.67:44485)
0925 15:20:10.952586 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:10.952593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:10.952620 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:20:10.952682 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:10.952688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:10.952897 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:10.953106 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:10.953463 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.954486 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.955630 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:10.956158 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:10.956248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:10.957270 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:10.957277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:10.957290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:10.957842 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:10.957886 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:10.957914 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:10.957997 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:10.958112 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:20:10.958249 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
I20260925 15:20:10.959852 18868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:11.039501 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18746
I20260925 15:20:11.055279 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18396
I20260925 15:20:11.056383 18457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:11.153752 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18396
2026-09-25T15:20:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2277 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:11Z Disabled control of system clock
I20260925 15:20:11.220242 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39955
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:20:11.523289 18892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:11.523545 18892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:11.523633 18892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:11.530421 18892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:11.530511 18892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:11.530550 18892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:11.530602 18892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:11.530676 18892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:11.538904 18892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:39955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:11.539989 18892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:11.541551 18892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:11.552457 18900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:11.552572 18897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:11.552546 18898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:11.553067 18892 server_base.cc:1034] running on GCE node
I20260925 15:20:11.553805 18892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:11.555128 18892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:11.556347 18892 hybrid_clock.cc:648] HybridClock initialized: now 1790349611556303 us; error 43 us; skew 500 ppm
I20260925 15:20:11.556672 18892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:11.558794 18892 webserver.cc:460] Webserver started at http://127.8.226.126:33743/ using document root <none> and password file <none>
I20260925 15:20:11.559492 18892 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:11.559620 18892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:11.559929 18892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:11.562690 18892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "1e05eb5c09614db2a4a1bd7acaf074d8"
format_stamp: "Formatted at 2026-09-25 15:20:11 on dist-test-slave-8vw7"
server_key: "689e3fbae151afbb840947102445ca7f"
server_key_iv: "ab4fef6e612d5d50cec5c820036bd5d3"
server_key_version: "encryptionkey@0"
I20260925 15:20:11.563421 18892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "1e05eb5c09614db2a4a1bd7acaf074d8"
format_stamp: "Formatted at 2026-09-25 15:20:11 on dist-test-slave-8vw7"
server_key: "689e3fbae151afbb840947102445ca7f"
server_key_iv: "ab4fef6e612d5d50cec5c820036bd5d3"
server_key_version: "encryptionkey@0"
I20260925 15:20:11.568027 18892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:20:11.571236 18906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:11.572445 18892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:11.572606 18892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "1e05eb5c09614db2a4a1bd7acaf074d8"
format_stamp: "Formatted at 2026-09-25 15:20:11 on dist-test-slave-8vw7"
server_key: "689e3fbae151afbb840947102445ca7f"
server_key_iv: "ab4fef6e612d5d50cec5c820036bd5d3"
server_key_version: "encryptionkey@0"
I20260925 15:20:11.572804 18892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:11.578305 18892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:11.579034 18892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:11.579339 18892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:11.596429 18892 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:39955
I20260925 15:20:11.596472 18957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:39955 every 8 connection(s)
I20260925 15:20:11.597702 18892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:11.598349 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18892
I20260925 15:20:11.598587 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:20:11.599316 9097 external_mini_cluster.cc:1428] Setting key 42b41590cb7b8591ae236d3a0e6fe055
I20260925 15:20:11.602041 18958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:11.608450 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:11.601214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.8.226.126:39955)
0925 15:20:11.602249 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:11.602262 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:11.602315 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:20:11.604538 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:11.604564 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:11.605128 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:11.605547 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:11.606514 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:11.606522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:11.606537 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:11.606552 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:11.606653 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:11.606674 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:11.606874 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:11.606995 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:20:11.607250 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":352,"threads_started":1}
I20260925 15:20:11.613953 18958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8: Bootstrap starting.
I20260925 15:20:11.616528 18958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:11.617533 18958 log.cc:824] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:11.619837 18958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8: No bootstrap required, opened a new log
I20260925 15:20:11.626547 18958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39955 } }
I20260925 15:20:11.626870 18958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:11.626983 18958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e05eb5c09614db2a4a1bd7acaf074d8, State: Initialized, Role: FOLLOWER
I20260925 15:20:11.627599 18958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39955 } }
I20260925 15:20:11.627889 18958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:11.628027 18958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:11.628211 18958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:11.630296 18958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39955 } }
I20260925 15:20:11.630784 18958 leader_election.cc:302] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e05eb5c09614db2a4a1bd7acaf074d8; no voters:
I20260925 15:20:11.631356 18958 leader_election.cc:288] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:11.631469 18963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:11.632237 18963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [term 1 LEADER]: Becoming Leader. State: Replica: 1e05eb5c09614db2a4a1bd7acaf074d8, State: Running, Role: LEADER
I20260925 15:20:11.632797 18963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39955 } }
I20260925 15:20:11.633490 18958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:11.635771 18965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e05eb5c09614db2a4a1bd7acaf074d8. Latest consensus state: current_term: 1 leader_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39955 } } }
I20260925 15:20:11.635962 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e05eb5c09614db2a4a1bd7acaf074d8" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39955 } } }
I20260925 15:20:11.636198 18965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:11.636276 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:11.637699 18972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:11.642521 18972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:11.651196 18972 catalog_manager.cc:1347] Generated new cluster ID: 0b6074bd6a654d5e8d758dd54b8564e1
I20260925 15:20:11.651307 18972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:11.674554 18972 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:11.674961 18972 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:11.682554 18972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e05eb5c09614db2a4a1bd7acaf074d8: Generated new TSK 0
I20260925 15:20:11.683279 18972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:11.689160 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39955
--builtin_ntp_servers=127.8.226.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:20:12.016462 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:12.016744 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:12.016834 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:12.023725 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:12.023815 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:12.023921 18982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:20:12.032568 18982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:12.033906 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:12.035820 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:12.048484 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.048527 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.048575 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.049299 18982 server_base.cc:1034] running on GCE node
I20260925 15:20:12.049894 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:12.050741 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:12.051976 18982 hybrid_clock.cc:648] HybridClock initialized: now 1790349612051934 us; error 44 us; skew 500 ppm
I20260925 15:20:12.052300 18982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:12.054502 18982 webserver.cc:460] Webserver started at http://127.8.226.65:42591/ using document root <none> and password file <none>
I20260925 15:20:12.055157 18982 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:12.055301 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:12.055636 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:12.058399 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "7e9d2a7dcaa84b4da2f144769a368e33"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "327a8ca56b0d1c8258dc21de6d446b53"
server_key_iv: "3876066389ec91f9c054c84959f32c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.059202 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "7e9d2a7dcaa84b4da2f144769a368e33"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "327a8ca56b0d1c8258dc21de6d446b53"
server_key_iv: "3876066389ec91f9c054c84959f32c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.063994 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 15:20:12.067121 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.068253 18982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:12.068413 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "7e9d2a7dcaa84b4da2f144769a368e33"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "327a8ca56b0d1c8258dc21de6d446b53"
server_key_iv: "3876066389ec91f9c054c84959f32c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.068594 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:12.074021 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:12.074743 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:12.075006 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:12.075891 18982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:12.077337 18982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:12.077450 18982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.077576 18982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:12.077666 18982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.108412 18982 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:35971
I20260925 15:20:12.108459 19108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:35971 every 8 connection(s)
I20260925 15:20:12.109833 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:12.113608 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 18982
I20260925 15:20:12.113816 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:12.114648 9097 external_mini_cluster.cc:1428] Setting key 1850a68f412736a872f60bf4476e4179
I20260925 15:20:12.117493 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39955
--builtin_ntp_servers=127.8.226.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:20:12.118661 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.112209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:52269 (local address 127.8.226.126:39955)
0925 15:20:12.112628 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.112632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.116146 (+ 3514us) server_negotiation.cc:407] Connection header received
0925 15:20:12.116234 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.116240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.116299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.116397 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.117800 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.117808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.117815 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:12.117836 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.117885 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:12.117902 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.118022 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.118181 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.118247 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":228,"threads_started":1}
I20260925 15:20:12.119417 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.112695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39955 (local address 127.8.226.65:52269)
0925 15:20:12.113811 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:12.113872 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:12.115861 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:12.116646 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:12.116655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:12.116740 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:12.117465 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:12.117467 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:12.117502 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:12.117612 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:12.117997 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:12.118004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:12.118010 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:12.118201 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:20:12.118376 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":288,"threads_started":1}
I20260925 15:20:12.120777 19109 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39955
I20260925 15:20:12.121070 19109 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:12.121727 19109 heartbeater.cc:511] Master 127.8.226.126:39955 requested a full tablet report, sending...
I20260925 15:20:12.123962 18923 ts_manager.cc:194] Registered new tserver with Master: 7e9d2a7dcaa84b4da2f144769a368e33 (127.8.226.65:35971)
W20260925 15:20:12.427925 19113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:12.428154 19113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:12.428231 19113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:12.435236 19113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:12.435324 19113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:12.435434 19113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:20:12.444315 19113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:12.445535 19113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:12.447125 19113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:12.459304 19118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.459223 19121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.459272 19119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.460350 19113 server_base.cc:1034] running on GCE node
I20260925 15:20:12.460878 19113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:12.461755 19113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:12.462972 19113 hybrid_clock.cc:648] HybridClock initialized: now 1790349612462942 us; error 43 us; skew 500 ppm
I20260925 15:20:12.463253 19113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:12.465397 19113 webserver.cc:460] Webserver started at http://127.8.226.66:34807/ using document root <none> and password file <none>
I20260925 15:20:12.465986 19113 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:12.466105 19113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:12.466398 19113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:12.470574 19113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "38e73e28283e42aaa559d28f71756e36"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "da84ef88230f2c5e6153022e90b3214f"
server_key_iv: "a8f2c6c82e09c94a0cc8f08c74885dea"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.471400 19113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "38e73e28283e42aaa559d28f71756e36"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "da84ef88230f2c5e6153022e90b3214f"
server_key_iv: "a8f2c6c82e09c94a0cc8f08c74885dea"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.476107 19113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:20:12.479174 19127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.480302 19113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:12.480472 19113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "38e73e28283e42aaa559d28f71756e36"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "da84ef88230f2c5e6153022e90b3214f"
server_key_iv: "a8f2c6c82e09c94a0cc8f08c74885dea"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.480688 19113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:12.485985 19113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:12.486681 19113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:12.486943 19113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:12.487748 19113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:12.489050 19113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:12.489156 19113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.489282 19113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:12.489375 19113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.520566 19113 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:35357
I20260925 15:20:12.520654 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:35357 every 8 connection(s)
I20260925 15:20:12.521854 19113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:12.529635 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.524247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:43735 (local address 127.8.226.126:39955)
0925 15:20:12.524416 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.524420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.525829 (+ 1409us) server_negotiation.cc:407] Connection header received
0925 15:20:12.527432 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.527438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.527510 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.527614 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.528833 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.528839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.528843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:12.528861 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.528899 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:12.528911 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.529070 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.529265 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.529354 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:20:12.530187 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 19113
I20260925 15:20:12.530382 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:12.530565 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.524762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39955 (local address 127.8.226.66:43735)
0925 15:20:12.525672 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:12.525706 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:12.527172 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:12.527822 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:12.527830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:12.527913 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:12.528552 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:12.528555 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:12.528588 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:12.528665 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:12.529044 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:12.529054 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:12.529061 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:20:12.529363 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 15:20:12.529591 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":304,"threads_started":1}
I20260925 15:20:12.530918 9097 external_mini_cluster.cc:1428] Setting key f0aec5a2092506744b792804ba990b65
I20260925 15:20:12.531842 19240 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39955
I20260925 15:20:12.532115 19240 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:12.532644 19240 heartbeater.cc:511] Master 127.8.226.126:39955 requested a full tablet report, sending...
I20260925 15:20:12.533774 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39955
--builtin_ntp_servers=127.8.226.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:20:12.534138 18923 ts_manager.cc:194] Registered new tserver with Master: 38e73e28283e42aaa559d28f71756e36 (127.8.226.66:35357)
W20260925 15:20:12.850607 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:12.850865 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:12.850976 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:12.858074 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:12.858182 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:12.858342 19243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:20:12.867012 19243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:12.868211 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:12.869961 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:12.882469 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.882427 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.882494 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.883208 19243 server_base.cc:1034] running on GCE node
I20260925 15:20:12.883896 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:12.884855 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:12.886071 19243 hybrid_clock.cc:648] HybridClock initialized: now 1790349612886011 us; error 49 us; skew 500 ppm
I20260925 15:20:12.886368 19243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:20:12.888435 19243 webserver.cc:460] Webserver started at http://127.8.226.67:33013/ using document root <none> and password file <none>
I20260925 15:20:12.889113 19243 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:12.889225 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:12.889542 19243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:12.892223 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "20679ecab8e244b8a57b28c0ef2ca63a"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "9f5ccf13ac093d4eb175c26e0417cc5e"
server_key_iv: "da9ac584a1ad48f53375701de5967731"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.893074 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "20679ecab8e244b8a57b28c0ef2ca63a"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "9f5ccf13ac093d4eb175c26e0417cc5e"
server_key_iv: "da9ac584a1ad48f53375701de5967731"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.897529 19243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:20:12.900642 19257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.901832 19243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:20:12.902029 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "20679ecab8e244b8a57b28c0ef2ca63a"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-8vw7"
server_key: "9f5ccf13ac093d4eb175c26e0417cc5e"
server_key_iv: "da9ac584a1ad48f53375701de5967731"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.902231 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:12.907353 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:12.908052 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:12.908285 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:12.909175 19243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:12.910425 19243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:12.910544 19243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.910665 19243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:12.910744 19243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.942399 19243 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:34271
I20260925 15:20:12.942492 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:34271 every 8 connection(s)
I20260925 15:20:12.943686 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:12.945078 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 19243
I20260925 15:20:12.945266 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:12.945859 9097 external_mini_cluster.cc:1428] Setting key b576e539862317649b5fe8442e3de674
I20260925 15:20:12.952229 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.946561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:51861 (local address 127.8.226.126:39955)
0925 15:20:12.946831 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.946837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.948457 (+ 1620us) server_negotiation.cc:407] Connection header received
0925 15:20:12.950295 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.950299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.950348 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.950429 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.951704 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.951708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.951711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:12.951724 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.951753 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:12.951764 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.951854 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.951978 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.952023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260925 15:20:12.953074 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.947230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39955 (local address 127.8.226.67:51861)
0925 15:20:12.948281 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:12.948331 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:12.950046 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:12.950683 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:12.950690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:12.950773 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:12.951436 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:12.951438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:12.951472 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:12.951547 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:12.951825 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:12.951829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:12.951833 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:12.952015 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 15:20:12.952187 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":357,"threads_started":1}
I20260925 15:20:12.954345 19370 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39955
I20260925 15:20:12.954623 19370 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:12.955166 19370 heartbeater.cc:511] Master 127.8.226.126:39955 requested a full tablet report, sending...
I20260925 15:20:12.956468 18923 ts_manager.cc:194] Registered new tserver with Master: 20679ecab8e244b8a57b28c0ef2ca63a (127.8.226.67:34271)
I20260925 15:20:12.961009 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:20:12.966475 19111 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:12.965112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.8.226.126:39955)
0925 15:20:12.965253 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.965257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.965720 (+ 463us) server_negotiation.cc:407] Connection header received
0925 15:20:12.965859 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.965863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.965913 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.965984 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.966254 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47904: BlockingRecv error: recv got EOF from 127.0.0.1:47904 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:20:12.967350 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18982
I20260925 15:20:12.970497 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.968159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.8.226.65:35971)
0925 15:20:12.968598 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.968604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.968628 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:20:12.968679 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.968684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.968854 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.969020 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.969862 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.969868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.969878 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:12.969890 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.969935 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:12.969948 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.970049 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.970117 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.970191 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":199,"threads_started":1}
I20260925 15:20:12.971560 19104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:13.053931 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18982
I20260925 15:20:13.070957 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 19113
I20260925 15:20:13.074522 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:13.071851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.8.226.66:35357)
0925 15:20:13.072382 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:13.072390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:13.072422 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:20:13.072500 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:13.072506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:13.072714 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:13.072882 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:13.073837 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:13.073843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:13.073854 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:13.073865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:13.073904 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:13.073917 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:13.074066 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:13.074186 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:20:13.074289 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":198,"threads_started":1}
I20260925 15:20:13.075780 19235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:13.155414 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 19113
I20260925 15:20:13.170413 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 19243
I20260925 15:20:13.174353 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:13.171301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.8.226.67:34271)
0925 15:20:13.171928 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:13.171935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:13.171964 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:20:13.172023 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:13.172029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:13.172238 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:13.172401 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:13.173590 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:13.173595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:13.173626 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:13.173642 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:13.173681 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:13.173694 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:13.173885 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:13.173959 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:20:13.174058 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":323,"threads_started":1}
I20260925 15:20:13.175534 19365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:13.255103 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 19243
I20260925 15:20:13.269688 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 18892
I20260925 15:20:13.270852 18953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:13.367404 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 18892
2026-09-25T15:20:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2213 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:13 dist-test-slave-8vw7 krb5kdc[19388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:13 dist-test-slave-8vw7 krb5kdc[19388](info): set up 2 sockets
Sep 25 15:20:13 dist-test-slave-8vw7 krb5kdc[19388](info): commencing operation
krb5kdc: starting...
W20260925 15:20:14.425932 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:20:14 dist-test-slave-8vw7 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:20:14.630241 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34167
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:34167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:14.941569 19401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:14.941825 19401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:14.941995 19401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:14.948743 19401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:14.948858 19401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:14.948953 19401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:14.949059 19401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:14.949141 19401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:14.958194 19401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:34167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:14.959358 19401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:14.960570 19401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:14.971765 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:14.971727 19406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:14.971796 19407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.972474 19401 server_base.cc:1034] running on GCE node
I20260925 15:20:14.973173 19401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:14.974342 19401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:14.975564 19401 hybrid_clock.cc:648] HybridClock initialized: now 1790349614975519 us; error 35 us; skew 500 ppm
I20260925 15:20:14.975909 19401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:20:14 dist-test-slave-8vw7 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:14.978863 19401 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:20:14.980118 19401 webserver.cc:460] Webserver started at http://127.8.226.126:43857/ using document root <none> and password file <none>
I20260925 15:20:14.980742 19401 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:14.980885 19401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:14.981209 19401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:14.983922 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "6b090eb0ca0946cca688737b1e39c6ec"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-8vw7"
server_key: "70f476eb825de07b252c1e5979f21d71"
server_key_iv: "e69d7e8d6b0b4242dcfda96fb527935d"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.984622 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "6b090eb0ca0946cca688737b1e39c6ec"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-8vw7"
server_key: "70f476eb825de07b252c1e5979f21d71"
server_key_iv: "e69d7e8d6b0b4242dcfda96fb527935d"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.989279 19401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 15:20:14.992316 19416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.993608 19401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:14.993784 19401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "6b090eb0ca0946cca688737b1e39c6ec"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-8vw7"
server_key: "70f476eb825de07b252c1e5979f21d71"
server_key_iv: "e69d7e8d6b0b4242dcfda96fb527935d"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.993963 19401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:15.008383 19401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:15.012327 19401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:15.012590 19401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:15.029770 19401 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:34167
I20260925 15:20:15.029800 19467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:34167 every 8 connection(s)
I20260925 15:20:15.030980 19401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:15.034320 19468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:15.039036 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 19401
I20260925 15:20:15.039273 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:20:15.040061 9097 external_mini_cluster.cc:1428] Setting key 5ade5cc1a877ca510f06347353d8375b
I20260925 15:20:15.048455 19468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec: Bootstrap starting.
I20260925 15:20:15.052093 19468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:15.053488 19468 log.cc:824] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:20:15 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:20:15.056267 19468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec: No bootstrap required, opened a new log
I20260925 15:20:15.062124 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:15.042194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.8.226.126:34167)
0925 15:20:15.042965 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:15.042975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:15.043013 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:20:15.044541 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:15.044562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:15.044916 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:15.045300 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:15.047030 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.048201 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.049936 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.050603 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.050727 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:15.055380 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:15.055387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:15.055400 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:15.055435 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:15.058426 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:15.059086 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:15.059092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:15.059097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:15.059269 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:15.059646 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:15.059651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:15.059654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:15.060104 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:15.060452 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:15.060815 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 15:20:15.061208 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":351,"threads_started":1}
I20260925 15:20:15.063730 19468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b090eb0ca0946cca688737b1e39c6ec" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34167 } }
I20260925 15:20:15.064090 19468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:15.064213 19468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b090eb0ca0946cca688737b1e39c6ec, State: Initialized, Role: FOLLOWER
I20260925 15:20:15.064795 19468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b090eb0ca0946cca688737b1e39c6ec" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34167 } }
I20260925 15:20:15.065057 19468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:15.065189 19468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:15.065357 19468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:15.067291 19468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b090eb0ca0946cca688737b1e39c6ec" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34167 } }
I20260925 15:20:15.067726 19468 leader_election.cc:302] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b090eb0ca0946cca688737b1e39c6ec; no voters:
I20260925 15:20:15.068264 19468 leader_election.cc:288] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:15.068378 19473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:15.069052 19473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [term 1 LEADER]: Becoming Leader. State: Replica: 6b090eb0ca0946cca688737b1e39c6ec, State: Running, Role: LEADER
I20260925 15:20:15.069553 19473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b090eb0ca0946cca688737b1e39c6ec" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34167 } }
I20260925 15:20:15.070221 19468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:15.072575 19474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b090eb0ca0946cca688737b1e39c6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b090eb0ca0946cca688737b1e39c6ec" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34167 } } }
I20260925 15:20:15.072619 19475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b090eb0ca0946cca688737b1e39c6ec. Latest consensus state: current_term: 1 leader_uuid: "6b090eb0ca0946cca688737b1e39c6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b090eb0ca0946cca688737b1e39c6ec" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34167 } } }
I20260925 15:20:15.072922 19475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:15.072922 19474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:15.074483 19481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:15.079900 19481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:15.087941 19481 catalog_manager.cc:1347] Generated new cluster ID: 75486ee32e7049eda8ff1f6b91ecc156
I20260925 15:20:15.088058 19481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:15.138023 19481 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:15.139226 19481 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:15.147792 19481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec: Generated new TSK 0
I20260925 15:20:15.148556 19481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:20:15.207986 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34167
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:15.518057 19494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:15.518311 19494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:15.518492 19494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:15.524922 19494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:15.525054 19494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:15.525242 19494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:20:15.533979 19494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:15.535122 19494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:15.536366 19494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:15.548563 19499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:15.548635 19500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:15.548597 19502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:15.549507 19494 server_base.cc:1034] running on GCE node
I20260925 15:20:15.550093 19494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:15.550956 19494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:15.552138 19494 hybrid_clock.cc:648] HybridClock initialized: now 1790349615552088 us; error 47 us; skew 500 ppm
I20260925 15:20:15.552369 19494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:20:15 dist-test-slave-8vw7 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:15.564433 19494 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:20:15.565809 19494 webserver.cc:460] Webserver started at http://127.8.226.65:35963/ using document root <none> and password file <none>
I20260925 15:20:15.566465 19494 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:15.566604 19494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:15.566962 19494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:15.569734 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "0441c732c0594b6ca38d4dccbb5af21c"
format_stamp: "Formatted at 2026-09-25 15:20:15 on dist-test-slave-8vw7"
server_key: "6c5f1275821b2eb8fc7445b9522c6c29"
server_key_iv: "094cce0f14d9862e7baba14de4bd4950"
server_key_version: "encryptionkey@0"
I20260925 15:20:15.570492 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "0441c732c0594b6ca38d4dccbb5af21c"
format_stamp: "Formatted at 2026-09-25 15:20:15 on dist-test-slave-8vw7"
server_key: "6c5f1275821b2eb8fc7445b9522c6c29"
server_key_iv: "094cce0f14d9862e7baba14de4bd4950"
server_key_version: "encryptionkey@0"
I20260925 15:20:15.575330 19494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:20:15.578464 19509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:15.579658 19494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:15.579823 19494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "0441c732c0594b6ca38d4dccbb5af21c"
format_stamp: "Formatted at 2026-09-25 15:20:15 on dist-test-slave-8vw7"
server_key: "6c5f1275821b2eb8fc7445b9522c6c29"
server_key_iv: "094cce0f14d9862e7baba14de4bd4950"
server_key_version: "encryptionkey@0"
I20260925 15:20:15.579998 19494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:15.598413 19494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:15.601970 19494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:15.602233 19494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:15.603089 19494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:15.604315 19494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:15.604382 19494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:15.604542 19494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:15.604602 19494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:15.635391 19494 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:33969
I20260925 15:20:15.635476 19621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:33969 every 8 connection(s)
I20260925 15:20:15.636620 19494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:15.637496 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 19494
I20260925 15:20:15.637713 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:15.638504 9097 external_mini_cluster.cc:1428] Setting key 4675385fa8310492d65e6f9378064603
Sep 25 15:20:15 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:20:15.656878 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:15.639459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:58741 (local address 127.8.226.126:34167)
0925 15:20:15.639920 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:15.639926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:15.640942 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 15:20:15.642667 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:15.642673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:15.642760 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:15.642869 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:15.644250 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.645481 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.646945 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.647315 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.647471 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:15.651307 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:15.651312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:15.651315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:15.651345 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:15.653845 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:15.654726 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:15.654730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:15.654732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:15.654865 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:15.655322 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:15.655325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:15.655327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:15.655562 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:15.655658 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:15.656477 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 15:20:15.656624 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260925 15:20:15.658370 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:15.640015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34167 (local address 127.8.226.65:58741)
0925 15:20:15.640783 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:15.640831 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:15.642406 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:15.643062 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:15.643069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:15.643543 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:15.644084 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:15.644100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.645608 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.645611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:15.646782 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:15.646792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.647548 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.647554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:15.647791 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:15.648467 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:15.648492 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:15.651117 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:15.653988 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:15.653999 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:15.654017 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:15.654618 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:15.654972 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:15.654979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:15.654985 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:15.655225 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:15.655682 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:15.655692 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:15.656131 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 15:20:15.656420 (+ 289us) client_negotiation.cc:240] Negotiation successful
0925 15:20:15.656998 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":344,"threads_started":1}
I20260925 15:20:15.659925 19622 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34167
I20260925 15:20:15.660188 19622 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:15.660919 19622 heartbeater.cc:511] Master 127.8.226.126:34167 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:20:15.662986 19433 ts_manager.cc:194] Registered new tserver with Master: 0441c732c0594b6ca38d4dccbb5af21c (127.8.226.65:33969)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
I20260925 15:20:15.664369 19433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:58741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:20:15.689553 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34167
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:15.990715 19628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:15.991046 19628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:15.991202 19628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:15.998194 19628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:15.998327 19628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:15.998502 19628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:20:16.007673 19628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:16.008919 19628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:16.010308 19628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:16.022944 19633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.023010 19634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.023448 19636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.023487 19628 server_base.cc:1034] running on GCE node
I20260925 15:20:16.024082 19628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:16.024928 19628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:16.026226 19628 hybrid_clock.cc:648] HybridClock initialized: now 1790349616026167 us; error 34 us; skew 500 ppm
I20260925 15:20:16.026502 19628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:20:16 dist-test-slave-8vw7 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:16.033531 19628 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:20:16.035038 19628 webserver.cc:460] Webserver started at http://127.8.226.66:36885/ using document root <none> and password file <none>
I20260925 15:20:16.035878 19628 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:16.036023 19628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:16.036317 19628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:16.039804 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "2b848ed0849d4082969de61659202fbd"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-8vw7"
server_key: "64627246b8dbec8354d7828d32acc837"
server_key_iv: "84563763a4bbbaa49455dfb68e7090fe"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.040606 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "2b848ed0849d4082969de61659202fbd"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-8vw7"
server_key: "64627246b8dbec8354d7828d32acc837"
server_key_iv: "84563763a4bbbaa49455dfb68e7090fe"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.046406 19628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:20:16.049770 19643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.050989 19628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:20:16.051105 19628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "2b848ed0849d4082969de61659202fbd"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-8vw7"
server_key: "64627246b8dbec8354d7828d32acc837"
server_key_iv: "84563763a4bbbaa49455dfb68e7090fe"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.051254 19628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:16.067457 19628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:16.071190 19628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:16.071416 19628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:16.072242 19628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:16.073493 19628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:16.073562 19628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.073640 19628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:16.073689 19628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.105253 19628 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:39573
I20260925 15:20:16.105294 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:39573 every 8 connection(s)
I20260925 15:20:16.106384 19628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:16.108877 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 19628
I20260925 15:20:16.109086 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:16.109580 9097 external_mini_cluster.cc:1428] Setting key 4e48586c92f1c6a97efda8a71886e21d
Sep 25 15:20:16 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:20:16.127605 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.108799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:57405 (local address 127.8.226.126:34167)
0925 15:20:16.108990 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.108995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.110357 (+ 1362us) server_negotiation.cc:407] Connection header received
0925 15:20:16.112284 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.112291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.112394 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.112519 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.114000 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.115142 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.116488 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.117171 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.117309 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.120676 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.120684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.120691 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.120730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.124337 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.125214 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.125221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.125228 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.125386 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.125776 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.125783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.125789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.126154 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.126327 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.126905 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.127150 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:20:16.128144 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.109428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34167 (local address 127.8.226.66:57405)
0925 15:20:16.110133 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.110200 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.112025 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.112701 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.112708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.113264 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.113806 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.113822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.115280 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.115283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.116359 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.116368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.117353 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.117357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.117558 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.118153 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.118177 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.120473 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.124499 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.124509 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.124527 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.125109 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.125491 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.125494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.125496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.125651 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.126270 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.126275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.126555 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.126761 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.127097 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":313,"threads_started":1}
I20260925 15:20:16.129563 19756 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34167
I20260925 15:20:16.129887 19756 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:16.130825 19756 heartbeater.cc:511] Master 127.8.226.126:34167 requested a full tablet report, sending...
I20260925 15:20:16.132325 19433 ts_manager.cc:194] Registered new tserver with Master: 2b848ed0849d4082969de61659202fbd (127.8.226.66:39573)
I20260925 15:20:16.133227 19433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:57405
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:20:16.160090 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34167
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:16.466877 19761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:16.467140 19761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:16.467223 19761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:16.474225 19761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:16.474349 19761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:16.474526 19761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:20:16.484302 19761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:16.485549 19761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:16.486836 19761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:16.499190 19766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.499213 19767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.499373 19769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.499974 19761 server_base.cc:1034] running on GCE node
I20260925 15:20:16.500515 19761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:16.501461 19761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:16.502645 19761 hybrid_clock.cc:648] HybridClock initialized: now 1790349616502594 us; error 48 us; skew 500 ppm
I20260925 15:20:16.502935 19761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:20:16 dist-test-slave-8vw7 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:16.505900 19761 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:20:16.507103 19761 webserver.cc:460] Webserver started at http://127.8.226.67:46349/ using document root <none> and password file <none>
I20260925 15:20:16.507680 19761 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:16.507776 19761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:16.508000 19761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:16.510715 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-8vw7"
server_key: "0290f5746efc1b2b445560fde9b04ca6"
server_key_iv: "a6c71f9b7635d63659a76ad3f4fffe88"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.511397 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-8vw7"
server_key: "0290f5746efc1b2b445560fde9b04ca6"
server_key_iv: "a6c71f9b7635d63659a76ad3f4fffe88"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.515916 19761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:20:16.519001 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.520198 19761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:20:16.520359 19761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-8vw7"
server_key: "0290f5746efc1b2b445560fde9b04ca6"
server_key_iv: "a6c71f9b7635d63659a76ad3f4fffe88"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.520540 19761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:16.549866 19761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:16.553479 19761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:16.553736 19761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:16.554744 19761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:16.556473 19761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:16.556543 19761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.556716 19761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:16.556771 19761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.588372 19761 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:35803
I20260925 15:20:16.588413 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:35803 every 8 connection(s)
I20260925 15:20:16.589645 19761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:16.599771 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 19761
I20260925 15:20:16.599977 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:16.600485 9097 external_mini_cluster.cc:1428] Setting key 28badf5e44d631016e7f4ad7c39a668c
Sep 25 15:20:16 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:20:16.608345 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.592079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:48933 (local address 127.8.226.126:34167)
0925 15:20:16.592294 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.592300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.593467 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 15:20:16.595163 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.595167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.595234 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.595318 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.596672 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.597569 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.599107 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.599489 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.599578 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.603536 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.603540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.603543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.603571 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.606073 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.606631 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.606634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.606636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.606752 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.607080 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.607082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.607084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.607320 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.607443 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.607975 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.608112 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:20:16.609251 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.592598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34167 (local address 127.8.226.67:48933)
0925 15:20:16.593319 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.593357 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.594893 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.595504 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.595511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.595978 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.596526 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.596541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.597693 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.597697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.598986 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.598997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.599697 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.599704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.600042 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.600710 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.600744 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.603342 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.606175 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.606181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.606190 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.606539 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.606837 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.606840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.606842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.606997 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.607425 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.607429 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.607697 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.607896 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.608198 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":263,"threads_started":1}
I20260925 15:20:16.610476 19889 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34167
I20260925 15:20:16.610745 19889 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:16.611418 19889 heartbeater.cc:511] Master 127.8.226.126:34167 requested a full tablet report, sending...
I20260925 15:20:16.612674 19433 ts_manager.cc:194] Registered new tserver with Master: d8fc8dfab82b4aebbb9ac597e5789d1f (127.8.226.67:35803)
I20260925 15:20:16.613458 19433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:48933
I20260925 15:20:16.615557 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:20:16.630548 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.619409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.8.226.126:34167)
0925 15:20:16.619537 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.619540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.620022 (+ 482us) server_negotiation.cc:407] Connection header received
0925 15:20:16.620133 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.620136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.620188 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.620263 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.621616 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.622261 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.623301 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.623667 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.623741 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.625855 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.625859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.625862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.625888 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.628099 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.628728 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.628733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.628737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.628856 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.629256 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.629261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.629264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.629563 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.629814 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.630097 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.630286 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:20:16.634029 19433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:16.642915 19433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:16.667111 19622 heartbeater.cc:503] Master 127.8.226.126:34167 was elected leader, sending a full tablet report...
I20260925 15:20:16.675800 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.654178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.8.226.67:35803)
0925 15:20:16.667509 (+ 13331us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.667521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.667561 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:20:16.667684 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.667694 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.668003 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.668972 (+ 969us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.669475 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.671116 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.673353 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.674707 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.674834 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.675075 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.675332 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.675474 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13116,"thread_start_us":13985,"threads_started":1}
I20260925 15:20:16.675865 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.653606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:35803 (local address 127.0.0.1:57076)
0925 15:20:16.660545 (+ 6939us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.660587 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.660773 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.668478 (+ 7705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.668490 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.668531 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.669296 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.669310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.671268 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.671273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.673174 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.673196 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.674883 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.674888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.675063 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.675121 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.675257 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.675422 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6665,"mutex_wait_us":32,"thread_start_us":6509,"threads_started":1}
I20260925 15:20:16.676007 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.653789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:33969 (local address 127.0.0.1:43284)
0925 15:20:16.667054 (+ 13265us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.667072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.667186 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.668567 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.668571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.668596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.669382 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.669396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.671339 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.671345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.673292 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.673309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.675271 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.675279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.675418 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.675450 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.675518 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.675636 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13159,"mutex_wait_us":43,"thread_start_us":15154,"threads_started":1}
I20260925 15:20:16.676584 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.654207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.8.226.65:33969)
0925 15:20:16.666009 (+ 11802us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.666021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.667308 (+ 1287us) server_negotiation.cc:407] Connection header received
0925 15:20:16.667403 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.667416 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.668279 (+ 863us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.668454 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.669597 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.671168 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.673478 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.674841 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.674957 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.675139 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.676180 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.676333 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11565,"thread_start_us":11812,"threads_started":1}
I20260925 15:20:16.677008 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.654188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:39573 (local address 127.0.0.1:54822)
0925 15:20:16.669384 (+ 15196us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.669400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.669522 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.670367 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.670372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.670394 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.670749 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.670759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.672614 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.672619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.674535 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.674545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.676242 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.676247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.676473 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.676504 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.676581 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.676696 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15032,"mutex_wait_us":48,"thread_start_us":12607,"threads_started":1}
I20260925 15:20:16.677726 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.654730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.8.226.66:39573)
0925 15:20:16.666614 (+ 11884us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.666626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.669625 (+ 2999us) server_negotiation.cc:407] Connection header received
0925 15:20:16.669738 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.669747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.669944 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.670119 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.670902 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.672475 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.674671 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.676013 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.676995 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.677169 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.677232 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.677369 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11728,"thread_start_us":11616,"threads_started":1}
I20260925 15:20:16.680431 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee (DEFAULT_TABLE table=test-table [id=6be2a6fb779943d896a9a84c12a269b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.680431 19557 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee (DEFAULT_TABLE table=test-table [id=6be2a6fb779943d896a9a84c12a269b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.680622 19691 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee (DEFAULT_TABLE table=test-table [id=6be2a6fb779943d896a9a84c12a269b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.681748 19691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.681777 19557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.681790 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.692624 19907 tablet_bootstrap.cc:492] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Bootstrap starting.
I20260925 15:20:16.695341 19909 tablet_bootstrap.cc:492] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Bootstrap starting.
I20260925 15:20:16.696062 19908 tablet_bootstrap.cc:492] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Bootstrap starting.
I20260925 15:20:16.697932 19907 tablet_bootstrap.cc:654] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.698952 19909 tablet_bootstrap.cc:654] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.699151 19907 log.cc:824] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:16.699338 19908 tablet_bootstrap.cc:654] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.700079 19909 log.cc:824] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:16.700452 19908 log.cc:824] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:16.702208 19907 tablet_bootstrap.cc:492] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: No bootstrap required, opened a new log
I20260925 15:20:16.702474 19907 ts_tablet_manager.cc:1397] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 15:20:16.702855 19908 tablet_bootstrap.cc:492] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: No bootstrap required, opened a new log
I20260925 15:20:16.702960 19909 tablet_bootstrap.cc:492] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: No bootstrap required, opened a new log
I20260925 15:20:16.703131 19908 ts_tablet_manager.cc:1397] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260925 15:20:16.703205 19909 ts_tablet_manager.cc:1397] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:20:16.709995 19908 raft_consensus.cc:348] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.710331 19908 raft_consensus.cc:374] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.710455 19908 raft_consensus.cc:729] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fc8dfab82b4aebbb9ac597e5789d1f, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.711021 19908 consensus_queue.cc:260] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.712006 19889 heartbeater.cc:503] Master 127.8.226.126:34167 was elected leader, sending a full tablet report...
I20260925 15:20:16.712498 19908 ts_tablet_manager.cc:1428] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Time spent starting tablet: real 0.009s user 0.007s sys 0.003s
I20260925 15:20:16.713124 19907 raft_consensus.cc:348] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.713510 19907 raft_consensus.cc:374] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.714111 19909 raft_consensus.cc:348] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.714423 19909 raft_consensus.cc:374] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.714558 19909 raft_consensus.cc:729] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0441c732c0594b6ca38d4dccbb5af21c, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.715129 19909 consensus_queue.cc:260] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.716627 19909 ts_tablet_manager.cc:1428] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:20:16.717239 19907 raft_consensus.cc:729] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b848ed0849d4082969de61659202fbd, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.717816 19907 consensus_queue.cc:260] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.718952 19756 heartbeater.cc:503] Master 127.8.226.126:34167 was elected leader, sending a full tablet report...
I20260925 15:20:16.719434 19907 ts_tablet_manager.cc:1428] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260925 15:20:16.841363 19890 tablet.cc:2369] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:16.858174 19757 tablet.cc:2369] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:16.879719 19914 raft_consensus.cc:482] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:16.880088 19914 raft_consensus.cc:504] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.881835 19914 leader_election.cc:288] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8fc8dfab82b4aebbb9ac597e5789d1f (127.8.226.67:35803), 2b848ed0849d4082969de61659202fbd (127.8.226.66:39573)
W20260925 15:20:16.888722 19623 tablet.cc:2369] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:16.890888 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.882388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:55417 (local address 127.8.226.66:39573)
0925 15:20:16.882578 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.882585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.883294 (+ 709us) server_negotiation.cc:407] Connection header received
0925 15:20:16.883498 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.883504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.883582 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.883698 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.884509 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.885612 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.888557 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.889773 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.889930 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.890000 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.890496 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.890622 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:20:16.890888 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.882388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:54729 (local address 127.8.226.67:35803)
0925 15:20:16.882578 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.882585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.883294 (+ 709us) server_negotiation.cc:407] Connection header received
0925 15:20:16.883498 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.883504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.883582 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.883694 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.884507 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.885627 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.888557 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.889775 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.889939 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.890000 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.890496 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.890623 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:20:16.891165 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.882194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:39573 (local address 127.8.226.65:55417)
0925 15:20:16.883163 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.883177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.883352 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.883717 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.883723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.883755 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.884322 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.884341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.885758 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.885761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.888401 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.888415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.890041 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.890046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.890242 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.890340 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.890452 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.890684 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":310,"spinlock_wait_cycles":20608,"thread_start_us":602,"threads_started":1}
I20260925 15:20:16.891191 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.882109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:35803 (local address 127.8.226.65:54729)
0925 15:20:16.883155 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.883177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.883352 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.883736 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.883739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.883764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.884354 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.884380 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.885756 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.885761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.888402 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.888415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.890019 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.890031 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.890242 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.890338 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.890452 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.890683 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":294,"spinlock_wait_cycles":21888,"thread_start_us":661,"threads_started":1}
I20260925 15:20:16.892396 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfcf40bc5c34fb78d0303e0cb8189ee" candidate_uuid: "0441c732c0594b6ca38d4dccbb5af21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" is_pre_election: true
I20260925 15:20:16.892398 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfcf40bc5c34fb78d0303e0cb8189ee" candidate_uuid: "0441c732c0594b6ca38d4dccbb5af21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b848ed0849d4082969de61659202fbd" is_pre_election: true
I20260925 15:20:16.892998 19844 raft_consensus.cc:2454] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0441c732c0594b6ca38d4dccbb5af21c in term 0.
I20260925 15:20:16.892993 19711 raft_consensus.cc:2454] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0441c732c0594b6ca38d4dccbb5af21c in term 0.
I20260925 15:20:16.893740 19512 leader_election.cc:302] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0441c732c0594b6ca38d4dccbb5af21c, 2b848ed0849d4082969de61659202fbd, d8fc8dfab82b4aebbb9ac597e5789d1f; no voters:
I20260925 15:20:16.894258 19914 raft_consensus.cc:2781] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:16.894404 19914 raft_consensus.cc:482] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:16.894495 19914 raft_consensus.cc:3037] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.896516 19914 raft_consensus.cc:504] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.897104 19914 leader_election.cc:288] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [CANDIDATE]: Term 1 election: Requested vote from peers d8fc8dfab82b4aebbb9ac597e5789d1f (127.8.226.67:35803), 2b848ed0849d4082969de61659202fbd (127.8.226.66:39573)
I20260925 15:20:16.897480 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfcf40bc5c34fb78d0303e0cb8189ee" candidate_uuid: "0441c732c0594b6ca38d4dccbb5af21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b848ed0849d4082969de61659202fbd"
I20260925 15:20:16.897459 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfcf40bc5c34fb78d0303e0cb8189ee" candidate_uuid: "0441c732c0594b6ca38d4dccbb5af21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f"
I20260925 15:20:16.897658 19844 raft_consensus.cc:3037] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.897653 19711 raft_consensus.cc:3037] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.899698 19711 raft_consensus.cc:2454] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0441c732c0594b6ca38d4dccbb5af21c in term 1.
I20260925 15:20:16.899899 19844 raft_consensus.cc:2454] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0441c732c0594b6ca38d4dccbb5af21c in term 1.
I20260925 15:20:16.900085 19513 leader_election.cc:302] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0441c732c0594b6ca38d4dccbb5af21c, 2b848ed0849d4082969de61659202fbd; no voters:
I20260925 15:20:16.900357 19914 raft_consensus.cc:2781] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:16.900835 19914 raft_consensus.cc:686] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 1 LEADER]: Becoming Leader. State: Replica: 0441c732c0594b6ca38d4dccbb5af21c, State: Running, Role: LEADER
I20260925 15:20:16.901374 19914 consensus_queue.cc:237] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } }
I20260925 15:20:16.905503 19433 catalog_manager.cc:5495] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0441c732c0594b6ca38d4dccbb5af21c (127.8.226.65). New cstate: current_term: 1 leader_uuid: "0441c732c0594b6ca38d4dccbb5af21c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0441c732c0594b6ca38d4dccbb5af21c" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 33969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:16.969697 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.964483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.8.226.65:33969)
0925 15:20:16.964691 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.964697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.964715 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:20:16.964935 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.964939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.965025 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.965171 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:16.965557 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.966279 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.967583 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.967996 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.968127 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.968619 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:16.969136 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:16.969301 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.969356 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.969431 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:20:16.977594 19711 raft_consensus.cc:1261] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 1 FOLLOWER]: Refusing update from remote peer 0441c732c0594b6ca38d4dccbb5af21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.977594 19844 raft_consensus.cc:1261] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 1 FOLLOWER]: Refusing update from remote peer 0441c732c0594b6ca38d4dccbb5af21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.978649 19918 consensus_queue.cc:1035] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8fc8dfab82b4aebbb9ac597e5789d1f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.978995 19914 consensus_queue.cc:1035] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b848ed0849d4082969de61659202fbd" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.991825 19923 mvcc.cc:205] Tried to move back new op lower bound from 7333272031129739264 to 7333272030831652864. Current Snapshot: MvccSnapshot[applied={T|T < 7333272031129739264}]
I20260925 15:20:16.998329 19922 mvcc.cc:205] Tried to move back new op lower bound from 7333272031129739264 to 7333272030831652864. Current Snapshot: MvccSnapshot[applied={T|T < 7333272031129739264}]
I20260925 15:20:17.005215 19432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:17.005548 19432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:17.009970 19432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b090eb0ca0946cca688737b1e39c6ec: Sending DeleteTablet for 3 replicas of tablet 9cfcf40bc5c34fb78d0303e0cb8189ee
I20260925 15:20:17.011031 19824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:17 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:57076
I20260925 15:20:17.011273 19557 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:17 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:43284
I20260925 15:20:17.011754 19691 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:17 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:54822
I20260925 15:20:17.012279 19690 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:17 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:54822
I20260925 15:20:17.014073 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 19494
I20260925 15:20:17.023584 19933 tablet_replica.cc:330] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: stopping tablet replica
I20260925 15:20:17.024116 19933 raft_consensus.cc:2229] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:17.024442 19931 tablet_replica.cc:330] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: stopping tablet replica
I20260925 15:20:17.024629 19933 raft_consensus.cc:2258] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:17.025285 19931 raft_consensus.cc:2229] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:17.025852 19931 raft_consensus.cc:2258] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:17.026731 19933 ts_tablet_manager.cc:1905] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:17.027942 19931 ts_tablet_manager.cc:1905] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:17.035974 19931 ts_tablet_manager.cc:1918] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:17.036199 19931 log.cc:1196] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/9cfcf40bc5c34fb78d0303e0cb8189ee
I20260925 15:20:17.036819 19931 ts_tablet_manager.cc:1939] T 9cfcf40bc5c34fb78d0303e0cb8189ee P d8fc8dfab82b4aebbb9ac597e5789d1f: Deleting consensus metadata
I20260925 15:20:17.038806 19419 catalog_manager.cc:4841] TS d8fc8dfab82b4aebbb9ac597e5789d1f (127.8.226.67:35803): tablet 9cfcf40bc5c34fb78d0303e0cb8189ee (table test-table [id=6be2a6fb779943d896a9a84c12a269b3]) successfully deleted
I20260925 15:20:17.049846 19933 ts_tablet_manager.cc:1918] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:17.050012 19933 log.cc:1196] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/9cfcf40bc5c34fb78d0303e0cb8189ee
I20260925 15:20:17.050525 19933 ts_tablet_manager.cc:1939] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 2b848ed0849d4082969de61659202fbd: Deleting consensus metadata
I20260925 15:20:17.051666 19932 tablet_replica.cc:330] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: stopping tablet replica
I20260925 15:20:17.052109 19932 raft_consensus.cc:2229] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:17.052232 19420 catalog_manager.cc:4841] TS 2b848ed0849d4082969de61659202fbd (127.8.226.66:39573): tablet 9cfcf40bc5c34fb78d0303e0cb8189ee (table test-table [id=6be2a6fb779943d896a9a84c12a269b3]) successfully deleted
I20260925 15:20:17.052873 19932 raft_consensus.cc:2258] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 15:20:17 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
W20260925 15:20:17.053194 19420 catalog_manager.cc:4808] TS 2b848ed0849d4082969de61659202fbd (127.8.226.66:39573): delete failed for tablet 9cfcf40bc5c34fb78d0303e0cb8189ee because the tablet was not found. No further retry: Not found: Tablet not found: 9cfcf40bc5c34fb78d0303e0cb8189ee
I20260925 15:20:17.055065 19932 ts_tablet_manager.cc:1905] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:17.062531 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:17.016689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.8.226.65:33969)
0925 15:20:17.016890 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:17.016899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:17.041608 (+ 24709us) server_negotiation.cc:407] Connection header received
0925 15:20:17.042009 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:17.042016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:17.042132 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:17.042259 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:17.045073 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:17.046349 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:17.047527 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:17.048390 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:17.048526 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:17.055000 (+ 6474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:17.055011 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:17.055027 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:17.055071 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:17.059031 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:17.059934 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:17.059950 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:17.059960 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:17.060168 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:17.060596 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:17.060608 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:17.060611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:17.061099 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:17.061408 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:17.061923 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 15:20:17.062144 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:20:17.062506 19932 ts_tablet_manager.cc:1918] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:17.062927 19932 log.cc:1196] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/9cfcf40bc5c34fb78d0303e0cb8189ee
I20260925 15:20:17.063653 19932 ts_tablet_manager.cc:1939] T 9cfcf40bc5c34fb78d0303e0cb8189ee P 0441c732c0594b6ca38d4dccbb5af21c: Deleting consensus metadata
I20260925 15:20:17.064209 19617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:17.177450 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 19494
I20260925 15:20:17.179694 19418 catalog_manager.cc:4841] TS 0441c732c0594b6ca38d4dccbb5af21c (127.8.226.65:33969): tablet 9cfcf40bc5c34fb78d0303e0cb8189ee (table test-table [id=6be2a6fb779943d896a9a84c12a269b3]) successfully deleted
I20260925 15:20:17.195281 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 19628
Sep 25 15:20:17 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:20:17.209244 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:17.195948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.8.226.66:39573)
0925 15:20:17.196127 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:17.196131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:17.196147 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:20:17.196268 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:17.196271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:17.196327 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:17.196410 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:17.197607 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:17.198477 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:17.199676 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:17.200035 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:17.200121 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:17.204067 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:17.204080 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:17.204093 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:17.204121 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:17.207020 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:17.207590 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:17.207597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:17.207599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:17.207701 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:17.207978 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:17.207981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:17.207983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:17.208318 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:17.208509 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:17.208753 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:20:17.208887 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:20:17.209900 19751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:17.296496 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 19628
I20260925 15:20:17.312563 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 19761
Sep 25 15:20:17 dist-test-slave-8vw7 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:20:17.326308 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:17.313355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.8.226.67:35803)
0925 15:20:17.313555 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:17.313560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:17.313577 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:20:17.313691 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:17.313695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:17.313763 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:17.313840 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:17.315015 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:17.315664 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:17.316764 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:17.317208 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:17.317292 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:17.321139 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:17.321152 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:17.321168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:17.321206 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:17.323626 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:17.324202 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:17.324212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:17.324214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:17.324351 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:17.324715 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:17.324721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:17.324726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:17.325219 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:17.325515 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:17.325788 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:20:17.325995 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:20:17.326900 19884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:17.411753 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 19761
I20260925 15:20:17.426999 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 19401
I20260925 15:20:17.428059 19463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:20:17.534807 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 19401
2026-09-25T15:20:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4169 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:17 dist-test-slave-8vw7 krb5kdc[19941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:17 dist-test-slave-8vw7 krb5kdc[19941](info): set up 2 sockets
Sep 25 15:20:17 dist-test-slave-8vw7 krb5kdc[19941](info): commencing operation
krb5kdc: starting...
W20260925 15:20:18.597755 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:20:18 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/rangeradmin_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/rangeradmin_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/rangerlookup_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/rangerlookup_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
I20260925 15:20:18.883495 9097 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskN5mqX7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/postgres -l logfile start
2026-09-25 15:20:21.182 UTC [19971] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:20:21.182 UTC [19971] LOG: listening on IPv4 address "127.8.226.84", port 53729
2026-09-25 15:20:21.185 UTC [19971] LOG: listening on Unix socket "/tmp/.s.PGSQL.53729"
2026-09-25 15:20:21.206 UTC [19974] LOG: database system was shut down at 2026-09-25 15:20:19 UTC
2026-09-25 15:20:21.209 UTC [19971] LOG: database system is ready to accept connections
I20260925 15:20:22.149566 9097 mini_postgres.cc:96] Postgres bound to 53729
2026-09-25 15:20:22.165 UTC [19982] FATAL: database "slave" does not exist
127.8.226.84:53729 - accepting connections
I20260925 15:20:22.166565 9097 mini_ranger.cc:161] Starting Ranger...
I20260925 15:20:22.190239 9097 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:20:22.630580 9097 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:20:22.630722 9097 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-admin
2026-09-25 15:20:22,935 [I] DB FLAVOR :POSTGRES
2026-09-25 15:20:22,936 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:20:22,936 [I] Checking connection..
2026-09-25 15:20:22,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:20:23,169 [I] Checking connection passed.
2026-09-25 15:20:23,170 [I] --------- Verifying version history table ---------
2026-09-25 15:20:23,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:20:23,421 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:20:23,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:20:23,660 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:20:23,660 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:20:23,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:20:23.894 UTC [20046] WARNING: there is no transaction in progress
2026-09-25 15:20:23,900 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:20:23,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:20:24,139 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:20:24,140 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:20:24,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:20:24,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:20:24,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8vw7.c.gcp-upstream.internal','N') ;"
2026-09-25 15:20:24,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:20:25,055 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:20:25,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:20:25,275 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:20:25,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:20:25,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:20:25,722 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:20:25,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:20:26.613 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.628 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.642 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.656 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.671 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.687 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.748 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.754 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.767 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.777 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.786 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.798 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.807 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.812 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.818 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:26.824 UTC [20154] WARNING: there is no transaction in progress
2026-09-25 15:20:27,398 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:20:27,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:20:27,635 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:20:27,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:20:27,871 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:20:27,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:20:28,111 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:20:28,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:20:28,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:20:28,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8vw7.c.gcp-upstream.internal';"
2026-09-25 15:20:28,821 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:20:28,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:20:29,044 [I] DB_PATCHES have already been applied
I20260925 15:20:29.051206 9097 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-admin:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskN5mqX7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:20:29.051288 9097 mini_ranger.cc:193] Using host: 127.8.226.84
I20260925 15:20:29.065606 9097 mini_ranger.cc:227] Ranger admin URL: http://127.8.226.84:53643
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:20:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:20:30.152784 9097 mini_ranger.cc:160] Time spent starting Ranger: real 7.986s user 0.001s sys 0.034s
Sep 25, 2026 3:20:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:20:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 25, 2026 3:20:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:20:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:20:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:20:49.808779 9097 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/rangerkms_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/rangerkms_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.226.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.226.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:20:49 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349649, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:20:49.980098 9097 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:20:50.001258 9097 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:20:50.783459 9097 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:20:50.783613 9097 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms
I20260925 15:20:50.783735 9097 mini_ranger_kms.cc:227] Using postgres at 127.8.226.84:53729
2026-09-25 15:20:51,190 [I] DB FLAVOR :POSTGRES
2026-09-25 15:20:51,190 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:20:51,190 [I] Checking connection
2026-09-25 15:20:51,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:20:51,436 [I] connection success
2026-09-25 15:20:51,436 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:20:51,436 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:20:51,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:20:51,669 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:20:51,670 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:20:51,670 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:20:51,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:53729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:20:51,916 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:20:52.385679 9097 mini_ranger_kms.cc:310] Created kms service
I20260925 15:20:52.587014 9097 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:20:52.630867 9097 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:20:52 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349649, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:20:53.025138 9097 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:20:53.025476 9097 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms:/tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:20:53.025542 9097 mini_ranger_kms.cc:240] Using host: 127.8.226.84
I20260925 15:20:53.047645 9097 mini_ranger_kms.cc:276] Ranger KMS PID: 20334
I20260925 15:20:53.047842 9097 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.226.84:59737
Sep 25, 2026 3:20:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:20:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:20:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:20:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:20:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:20:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 15:20:55 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349655, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.226.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:55.221688 9097 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.242s user 0.101s sys 0.322s
I20260925 15:20:55.221958 9097 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:20:55.222194 9097 mini_ranger_kms.cc:401] 127.8.226.84:59737/kms/v1/keys
Sep 25 15:20:55 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349655, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.226.84@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
Sep 25 15:20:55 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349655, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.226.84@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
Sep 25 15:20:58 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:20:58.109025 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:36955
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:36955
--ranger_config_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:59737/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:58.426415 20385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:58.426708 20385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:58.426870 20385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:58.433997 20385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:58.434092 20385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:58.434191 20385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:58.434302 20385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:58.434386 20385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:58.443360 20385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.226.84:59737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:36955
--ranger_config_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:58.444579 20385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:58.445864 20385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:58.457983 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:58.458038 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:58.458019 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:58.458717 20385 server_base.cc:1034] running on GCE node
I20260925 15:20:58.459466 20385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:58.460723 20385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:58.461927 20385 hybrid_clock.cc:648] HybridClock initialized: now 1790349658461885 us; error 39 us; skew 500 ppm
I20260925 15:20:58.462236 20385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:20:58 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:58.471026 20385 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:20:58.472371 20385 webserver.cc:460] Webserver started at http://127.8.226.126:36757/ using document root <none> and password file <none>
I20260925 15:20:58.472931 20385 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:58.473104 20385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:58.473387 20385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:20:58 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:20:58.525660 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487"
format_stamp: "Formatted at 2026-09-25 15:20:58 on dist-test-slave-8vw7"
server_key: "129b5cd587e3373cc6cea258d01e15be"
server_key_iv: "25d514489deb2be6695ae6bd1a9e5d85"
server_key_version: "kuduclusterkey@0"
I20260925 15:20:58.526471 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487"
format_stamp: "Formatted at 2026-09-25 15:20:58 on dist-test-slave-8vw7"
server_key: "129b5cd587e3373cc6cea258d01e15be"
server_key_iv: "25d514489deb2be6695ae6bd1a9e5d85"
server_key_version: "kuduclusterkey@0"
I20260925 15:20:58.531581 20385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:20:58.546120 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:58.547533 20385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:58.547693 20385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487"
format_stamp: "Formatted at 2026-09-25 15:20:58 on dist-test-slave-8vw7"
server_key: "129b5cd587e3373cc6cea258d01e15be"
server_key_iv: "25d514489deb2be6695ae6bd1a9e5d85"
server_key_version: "kuduclusterkey@0"
I20260925 15:20:58.547895 20385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:58.565240 20385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:58.568866 20385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:58.569172 20385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:58.588675 20385 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:36955
I20260925 15:20:58.588706 20451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:36955 every 8 connection(s)
I20260925 15:20:58.590104 20385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:58.593133 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 20385
I20260925 15:20:58.593325 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:20:58.594054 9097 external_mini_cluster.cc:1428] Setting key 38b176ffadc91d16ece48872fa343f94
I20260925 15:20:58.594146 20452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:58.607849 20452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487: Bootstrap starting.
Sep 25 15:20:58 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:20:58.611167 20452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:58.612298 20452 log.cc:824] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:58.615171 20452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487: No bootstrap required, opened a new log
I20260925 15:20:58.620494 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:58.596248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.8.226.126:36955)
0925 15:20:58.597033 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:58.597046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:58.597093 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:20:58.598935 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:58.598961 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:58.599814 (+ 853us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:58.600236 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:58.601737 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.603563 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.605420 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.606370 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.606512 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.611157 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:58.611167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:58.611184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:58.611232 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:58.615703 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:58.616444 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:58.616451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:58.616458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:58.616639 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:58.617066 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:58.617071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:58.617074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:58.617805 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:58.618094 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:58.618582 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:20:58.619123 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":377,"threads_started":1}
I20260925 15:20:58.623857 20452 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36955 } }
I20260925 15:20:58.624190 20452 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:58.624291 20452 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8f3abb3cf14c48b3ac9cbce72ad487, State: Initialized, Role: FOLLOWER
I20260925 15:20:58.624866 20452 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36955 } }
I20260925 15:20:58.625138 20452 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:58.625277 20452 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:58.625438 20452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:58.627498 20452 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36955 } }
I20260925 15:20:58.627954 20452 leader_election.cc:302] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8f3abb3cf14c48b3ac9cbce72ad487; no voters:
I20260925 15:20:58.628548 20452 leader_election.cc:288] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:58.628638 20457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:58.631361 20452 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:58.635963 20457 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [term 1 LEADER]: Becoming Leader. State: Replica: fa8f3abb3cf14c48b3ac9cbce72ad487, State: Running, Role: LEADER
I20260925 15:20:58.636734 20457 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36955 } }
I20260925 15:20:58.639676 20458 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36955 } } }
I20260925 15:20:58.640051 20458 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:58.640585 20459 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa8f3abb3cf14c48b3ac9cbce72ad487. Latest consensus state: current_term: 1 leader_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8f3abb3cf14c48b3ac9cbce72ad487" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36955 } } }
I20260925 15:20:58.640890 20459 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:58.645834 20465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:58.650771 20465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:58.651527 20452 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:20:58.677486 20465 catalog_manager.cc:1347] Generated new cluster ID: 0dbc24e9ee964dfa8668956378143c58
I20260925 15:20:58.677989 20465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:58.698719 20465 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:58.700695 20465 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:58.716389 20465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487: Generated new TSK 0
I20260925 15:20:58.718657 20465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:20:59.468901 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:59737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36955
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:20:59 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:20:59.849644 20497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:59.849915 20497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:59.850057 20497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:59.857059 20497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:59.857175 20497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:59.857374 20497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:20:59.866647 20497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.226.84:59737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:20:59.868018 20497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:59.869485 20497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:59.886174 20507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:59.887151 20508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:59.895337 20510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:59.896384 20497 server_base.cc:1034] running on GCE node
I20260925 15:20:59.897060 20497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:59.898113 20497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:59.899386 20497 hybrid_clock.cc:648] HybridClock initialized: now 1790349659899330 us; error 52 us; skew 500 ppm
I20260925 15:20:59.899690 20497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:20:59 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:59.903227 20497 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:20:59.907042 20497 webserver.cc:460] Webserver started at http://127.8.226.65:40631/ using document root <none> and password file <none>
I20260925 15:20:59.907689 20497 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:59.907811 20497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:59.908114 20497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:20:59 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:20:59.935905 20497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "e5992d9666dd4f8a83cec6c685895cce"
format_stamp: "Formatted at 2026-09-25 15:20:59 on dist-test-slave-8vw7"
server_key: "26364dfb9c26511c06455b4d8853473d"
server_key_iv: "538e3298ab48c62e7c512f9fd3db093e"
server_key_version: "kuduclusterkey@0"
I20260925 15:20:59.939109 20497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "e5992d9666dd4f8a83cec6c685895cce"
format_stamp: "Formatted at 2026-09-25 15:20:59 on dist-test-slave-8vw7"
server_key: "26364dfb9c26511c06455b4d8853473d"
server_key_iv: "538e3298ab48c62e7c512f9fd3db093e"
server_key_version: "kuduclusterkey@0"
I20260925 15:20:59.959106 20497 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.011s sys 0.007s
I20260925 15:20:59.987839 20517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:59.989975 20497 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260925 15:20:59.990255 20497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "e5992d9666dd4f8a83cec6c685895cce"
format_stamp: "Formatted at 2026-09-25 15:20:59 on dist-test-slave-8vw7"
server_key: "26364dfb9c26511c06455b4d8853473d"
server_key_iv: "538e3298ab48c62e7c512f9fd3db093e"
server_key_version: "kuduclusterkey@0"
I20260925 15:20:59.990600 20497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:00.013075 20497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:00.018616 20497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:00.018987 20497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:00.026258 20497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:00.034219 20497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:00.034377 20497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:00.034557 20497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:00.034662 20497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:00.116632 20497 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:36707
I20260925 15:21:00.116658 20629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:36707 every 8 connection(s)
I20260925 15:21:00.118846 20497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:00.124795 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 20497
I20260925 15:21:00.125016 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:21:00.125800 9097 external_mini_cluster.cc:1428] Setting key 0c1c67d1b60c7b362c6f7167a2796d17
Sep 25 15:21:00 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:00.167220 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.123368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:35055 (local address 127.8.226.126:36955)
0925 15:21:00.145590 (+ 22222us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:00.145595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:00.145610 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:21:00.145655 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:00.145659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:00.145746 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:00.145878 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:00.149152 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.149939 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.151663 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.152293 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.152412 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.160083 (+ 7671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:00.160090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:00.160095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:00.160135 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:00.163492 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:00.164400 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:00.164408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:00.164414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:00.164561 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:00.165117 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:00.165122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:00.165126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:00.165491 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:00.165623 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:00.166584 (+ 961us) server_negotiation.cc:299] Negotiation successful
0925 15:21:00.166803 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22104,"thread_start_us":21964,"threads_started":1}
I20260925 15:21:00.169638 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.124062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36955 (local address 127.8.226.65:35055)
0925 15:21:00.141648 (+ 17586us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:00.141714 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:00.144787 (+ 3073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:00.146435 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:00.146447 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:00.147242 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:00.148329 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.148377 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.150097 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.150101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.151489 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.151509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.155890 (+ 4381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.155898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.156191 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.157052 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:00.157083 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:00.159826 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:00.163641 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:00.163651 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:00.163670 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:00.164243 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:00.164689 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:00.164694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:00.164698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:00.164946 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:00.165721 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:00.165729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:00.166138 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 15:21:00.167432 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0925 15:21:00.168095 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17215,"thread_start_us":16979,"threads_started":1}
I20260925 15:21:00.171453 20630 heartbeater.cc:347] Connected to a master server at 127.8.226.126:36955
I20260925 15:21:00.171769 20630 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:00.172670 20630 heartbeater.cc:511] Master 127.8.226.126:36955 requested a full tablet report, sending...
I20260925 15:21:00.175321 20417 ts_manager.cc:194] Registered new tserver with Master: e5992d9666dd4f8a83cec6c685895cce (127.8.226.65:36707)
I20260925 15:21:00.177457 20417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:35055
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:21:00.222417 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:59737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36955
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:21:00 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
Sep 25 15:21:00 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349659, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
W20260925 15:21:00.690568 20636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:00.691025 20636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:00.691224 20636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:00.702492 20636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:00.702695 20636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:00.702926 20636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:21:00.717963 20636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.226.84:59737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:00.719847 20636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:00.721544 20636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:00.748832 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:00.748770 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:00.751900 20636 server_base.cc:1034] running on GCE node
W20260925 15:21:00.755721 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:00.756454 20636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:00.757740 20636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:00.759030 20636 hybrid_clock.cc:648] HybridClock initialized: now 1790349660758964 us; error 53 us; skew 500 ppm
I20260925 15:21:00.759420 20636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:00 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349660, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:00.763538 20636 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:21:00.765071 20636 webserver.cc:460] Webserver started at http://127.8.226.66:38339/ using document root <none> and password file <none>
I20260925 15:21:00.765687 20636 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:00.765775 20636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:00.766076 20636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:00 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349660, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:00.781769 20636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "5cc5216405604ba6ae10ff1648736b05"
format_stamp: "Formatted at 2026-09-25 15:21:00 on dist-test-slave-8vw7"
server_key: "6e85c73fefe564af39e60a1b914690ca"
server_key_iv: "e56a464e173bf27de72c517d83c21f7d"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:00.782692 20636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc5216405604ba6ae10ff1648736b05"
format_stamp: "Formatted at 2026-09-25 15:21:00 on dist-test-slave-8vw7"
server_key: "6e85c73fefe564af39e60a1b914690ca"
server_key_iv: "e56a464e173bf27de72c517d83c21f7d"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:00.789474 20636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 15:21:00.807854 20660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:00.809690 20636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:21:00.809911 20636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "5cc5216405604ba6ae10ff1648736b05"
format_stamp: "Formatted at 2026-09-25 15:21:00 on dist-test-slave-8vw7"
server_key: "6e85c73fefe564af39e60a1b914690ca"
server_key_iv: "e56a464e173bf27de72c517d83c21f7d"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:00.810249 20636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:00.841713 20636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:00.845754 20636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:00.846081 20636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:00.847515 20636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:00.848919 20636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:00.849061 20636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:21:00.849208 20636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:00.849279 20636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:00.887542 20636 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:38151
I20260925 15:21:00.887569 20772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:38151 every 8 connection(s)
I20260925 15:21:00.888765 20636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:21:00.892614 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 20636
I20260925 15:21:00.892758 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:21:00.893433 9097 external_mini_cluster.cc:1428] Setting key 44afed15c5cf4e8513cc2031bb6cbae0
Sep 25 15:21:00 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349660, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:00.910297 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.891797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:37211 (local address 127.8.226.126:36955)
0925 15:21:00.892212 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:00.892217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:00.893319 (+ 1102us) server_negotiation.cc:407] Connection header received
0925 15:21:00.895528 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:00.895534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:00.895627 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:00.895734 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:00.897201 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.897874 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.899430 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.899831 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.899959 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.903155 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:00.903159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:00.903162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:00.903189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:00.906574 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:00.907399 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:00.907404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:00.907407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:00.907535 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:00.908097 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:00.908103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:00.908107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:00.908466 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:00.909339 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:00.909516 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:21:00.909800 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":198,"threads_started":1}
I20260925 15:21:00.911130 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.892372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36955 (local address 127.8.226.66:37211)
0925 15:21:00.893163 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:00.893205 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:00.895253 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:00.895948 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:00.895956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:00.896441 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:00.897042 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.897065 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.898022 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.898027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.899278 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.899295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.900037 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.900043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.900345 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.901063 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:00.901084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:00.902959 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:00.906747 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:00.906756 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:00.906771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:00.907268 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:00.907650 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:00.907655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:00.907658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:00.907938 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:00.908614 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:00.908622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:00.909021 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 15:21:00.909303 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 15:21:00.909824 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":330,"threads_started":1}
I20260925 15:21:00.912642 20773 heartbeater.cc:347] Connected to a master server at 127.8.226.126:36955
I20260925 15:21:00.912942 20773 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:00.913756 20773 heartbeater.cc:511] Master 127.8.226.126:36955 requested a full tablet report, sending...
I20260925 15:21:00.915326 20417 ts_manager.cc:194] Registered new tserver with Master: 5cc5216405604ba6ae10ff1648736b05 (127.8.226.66:38151)
I20260925 15:21:00.916541 20417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:37211
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:21:00.949011 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:59737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36955
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:21:01.180938 20630 heartbeater.cc:503] Master 127.8.226.126:36955 was elected leader, sending a full tablet report...
W20260925 15:21:01.270850 20779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:01.271144 20779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:01.271306 20779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:01.278884 20779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:01.279016 20779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:01.279181 20779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:21:01.288319 20779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.226.84:59737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:36955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:01.289565 20779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:01.290832 20779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:01.303375 20784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:01.303354 20785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:01.303483 20787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:01.304352 20779 server_base.cc:1034] running on GCE node
I20260925 15:21:01.304807 20779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:01.305637 20779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:01.306821 20779 hybrid_clock.cc:648] HybridClock initialized: now 1790349661306781 us; error 34 us; skew 500 ppm
I20260925 15:21:01.307106 20779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:01 dist-test-slave-8vw7 krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349661, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:01.309973 20779 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:21:01.311264 20779 webserver.cc:460] Webserver started at http://127.8.226.67:44881/ using document root <none> and password file <none>
I20260925 15:21:01.311813 20779 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:01.311916 20779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:01.312160 20779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:01 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349661, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:01.324924 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "85e984f3ed484d8c92237e081930b7a1"
format_stamp: "Formatted at 2026-09-25 15:21:01 on dist-test-slave-8vw7"
server_key: "7a47f9004f4f3e40df807faa4ed8c76b"
server_key_iv: "0c7f1506f3067b03a525538a0244ba40"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:01.325752 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "85e984f3ed484d8c92237e081930b7a1"
format_stamp: "Formatted at 2026-09-25 15:21:01 on dist-test-slave-8vw7"
server_key: "7a47f9004f4f3e40df807faa4ed8c76b"
server_key_iv: "0c7f1506f3067b03a525538a0244ba40"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:01.330806 20779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 15:21:01.345573 20794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:01.346992 20779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:21:01.347155 20779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "85e984f3ed484d8c92237e081930b7a1"
format_stamp: "Formatted at 2026-09-25 15:21:01 on dist-test-slave-8vw7"
server_key: "7a47f9004f4f3e40df807faa4ed8c76b"
server_key_iv: "0c7f1506f3067b03a525538a0244ba40"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:01.347368 20779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:01.388942 20779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:01.392287 20779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:01.392557 20779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:01.393414 20779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:01.394639 20779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:01.394731 20779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:01.394860 20779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:01.395351 20779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:01.427459 20779 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:35887
I20260925 15:21:01.427495 20906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:35887 every 8 connection(s)
I20260925 15:21:01.428704 20779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:01.432766 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 20779
I20260925 15:21:01.432937 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:21:01.433471 9097 external_mini_cluster.cc:1428] Setting key 506dd32a6565146af5aa558064f2ed41
Sep 25 15:21:01 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349661, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:01.448626 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.431282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:35705 (local address 127.8.226.126:36955)
0925 15:21:01.431770 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.431778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.432716 (+ 938us) server_negotiation.cc:407] Connection header received
0925 15:21:01.435063 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.435067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.435168 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.435300 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.436789 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.437504 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.438950 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.439385 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.439505 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.442925 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:01.442929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:01.442932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:01.442960 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:01.445620 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.446166 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.446169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.446171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.446288 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.446689 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.446691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.446693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.447025 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:01.447168 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.447981 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.448219 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
I20260925 15:21:01.449551 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.431766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36955 (local address 127.8.226.67:35705)
0925 15:21:01.432525 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.432593 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.434784 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.435547 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.435555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.435949 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:01.436661 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.436676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.437619 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.437622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.438800 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.438851 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.439582 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.439586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.439771 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.440776 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:01.440796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:01.442758 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:01.445718 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:01.445723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:01.445733 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:01.446080 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:01.446367 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:01.446370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:01.446372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:01.446597 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:01.447157 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:01.447162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:01.447560 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 15:21:01.447882 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 15:21:01.448369 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":277,"threads_started":1}
I20260925 15:21:01.450935 20907 heartbeater.cc:347] Connected to a master server at 127.8.226.126:36955
I20260925 15:21:01.451184 20907 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:01.451967 20907 heartbeater.cc:511] Master 127.8.226.126:36955 requested a full tablet report, sending...
I20260925 15:21:01.453280 20417 ts_manager.cc:194] Registered new tserver with Master: 85e984f3ed484d8c92237e081930b7a1 (127.8.226.67:35887)
I20260925 15:21:01.453997 20417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:35705
I20260925 15:21:01.460279 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:01.476047 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.464481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.8.226.126:36955)
0925 15:21:01.464668 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.464676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.465033 (+ 357us) server_negotiation.cc:407] Connection header received
0925 15:21:01.465185 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.465188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.465252 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.465340 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.466581 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.467218 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.468793 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.469274 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.469372 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.471657 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:01.471661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:01.471664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:01.471690 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:01.474092 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.474628 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.474631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.474634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.474742 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.475041 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.475044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.475045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.475262 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:01.475471 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.475648 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.475784 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:21:01.479768 20417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:21:01.486476 20417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:21:01.519457 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.498734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:36707 (local address 127.0.0.1:33428)
0925 15:21:01.499599 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.499650 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.499847 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.510820 (+ 10973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.510830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.510880 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:01.512046 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.512057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.514385 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.514390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.516512 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.516524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.518474 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.518479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.518674 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.518729 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:21:01.518872 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:21:01.519035 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":23,"thread_start_us":13313,"threads_started":1}
I20260925 15:21:01.519505 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.499513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.8.226.65:36707)
0925 15:21:01.510093 (+ 10580us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.510104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.510139 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:21:01.510266 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.510277 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.510559 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.510782 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:01.512294 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.514198 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.516690 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.518027 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.518136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.518341 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.518942 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.519096 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10348,"thread_start_us":11487,"threads_started":1}
I20260925 15:21:01.520766 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.498771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.8.226.66:38151)
0925 15:21:01.511685 (+ 12914us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.511695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.511724 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:21:01.512369 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.512377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.513015 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.513202 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:01.513858 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.515257 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.517837 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.519053 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.519165 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.519387 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.520321 (+ 934us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.520453 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12778,"thread_start_us":12704,"threads_started":1}
I20260925 15:21:01.521253 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.498763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:38151 (local address 127.0.0.1:57836)
0925 15:21:01.511278 (+ 12515us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.511294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.511926 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.513206 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.513213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.513245 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:01.513685 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.513697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.515416 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.515423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.517702 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.517714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.519537 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.519543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.519938 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.519987 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:21:01.520068 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:21:01.520175 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12386,"mutex_wait_us":25,"thread_start_us":966,"threads_started":1}
I20260925 15:21:01.523372 20565 tablet_service.cc:1467] Processing CreateTablet for tablet d24ed6d87f3e4251b7bcace90b398d28 (DEFAULT_TABLE table=test-table [id=c6722355e16d49fc9fba2bf5217ac5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:01.524766 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24ed6d87f3e4251b7bcace90b398d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:01.524869 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.501004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:35887 (local address 127.0.0.1:51734)
0925 15:21:01.512315 (+ 11311us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.512336 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.512461 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.517938 (+ 5477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.517943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.517965 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:01.518308 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.518319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.520090 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.520097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.522488 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.522523 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.524116 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.524122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.524316 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.524350 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:21:01.524446 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:21:01.524554 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11156,"thread_start_us":10249,"threads_started":1}
I20260925 15:21:01.524984 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.501346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.8.226.67:35887)
0925 15:21:01.517415 (+ 16069us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.517422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.517449 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:21:01.517532 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.517539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.517732 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.517934 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:01.518487 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.519947 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.522671 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.523924 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.524037 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.524237 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.524497 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.524640 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15906,"thread_start_us":15765,"threads_started":1}
I20260925 15:21:01.524876 20708 tablet_service.cc:1467] Processing CreateTablet for tablet d24ed6d87f3e4251b7bcace90b398d28 (DEFAULT_TABLE table=test-table [id=c6722355e16d49fc9fba2bf5217ac5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:01.526224 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24ed6d87f3e4251b7bcace90b398d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:01.528378 20842 tablet_service.cc:1467] Processing CreateTablet for tablet d24ed6d87f3e4251b7bcace90b398d28 (DEFAULT_TABLE table=test-table [id=c6722355e16d49fc9fba2bf5217ac5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:01.529772 20842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24ed6d87f3e4251b7bcace90b398d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:01.537842 20926 tablet_bootstrap.cc:492] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Bootstrap starting.
I20260925 15:21:01.541129 20926 tablet_bootstrap.cc:654] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:01.541675 20927 tablet_bootstrap.cc:492] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Bootstrap starting.
I20260925 15:21:01.542482 20926 log.cc:824] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:01.544118 20929 tablet_bootstrap.cc:492] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Bootstrap starting.
I20260925 15:21:01.545342 20927 tablet_bootstrap.cc:654] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:01.545608 20926 tablet_bootstrap.cc:492] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: No bootstrap required, opened a new log
I20260925 15:21:01.545852 20926 ts_tablet_manager.cc:1397] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 15:21:01.546824 20927 log.cc:824] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:01.548440 20929 tablet_bootstrap.cc:654] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:01.549592 20929 log.cc:824] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:01.551640 20927 tablet_bootstrap.cc:492] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: No bootstrap required, opened a new log
I20260925 15:21:01.551885 20927 ts_tablet_manager.cc:1397] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260925 15:21:01.552611 20929 tablet_bootstrap.cc:492] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: No bootstrap required, opened a new log
I20260925 15:21:01.552839 20929 ts_tablet_manager.cc:1397] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:21:01.554807 20926 raft_consensus.cc:348] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.555130 20926 raft_consensus.cc:374] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:01.555234 20926 raft_consensus.cc:729] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc5216405604ba6ae10ff1648736b05, State: Initialized, Role: FOLLOWER
I20260925 15:21:01.555800 20926 consensus_queue.cc:260] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.556778 20773 heartbeater.cc:503] Master 127.8.226.126:36955 was elected leader, sending a full tablet report...
I20260925 15:21:01.557331 20926 ts_tablet_manager.cc:1428] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260925 15:21:01.562974 20929 raft_consensus.cc:348] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.563277 20929 raft_consensus.cc:374] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:01.563380 20929 raft_consensus.cc:729] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e984f3ed484d8c92237e081930b7a1, State: Initialized, Role: FOLLOWER
I20260925 15:21:01.563840 20929 consensus_queue.cc:260] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.564764 20907 heartbeater.cc:503] Master 127.8.226.126:36955 was elected leader, sending a full tablet report...
I20260925 15:21:01.564620 20927 raft_consensus.cc:348] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.565059 20927 raft_consensus.cc:374] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:01.565157 20927 raft_consensus.cc:729] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5992d9666dd4f8a83cec6c685895cce, State: Initialized, Role: FOLLOWER
I20260925 15:21:01.565385 20929 ts_tablet_manager.cc:1428] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 15:21:01.565688 20927 consensus_queue.cc:260] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.567231 20927 ts_tablet_manager.cc:1428] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Time spent starting tablet: real 0.015s user 0.001s sys 0.012s
I20260925 15:21:01.640758 20934 raft_consensus.cc:482] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 15:21:01.640802 20774 tablet.cc:2369] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:01.641456 20934 raft_consensus.cc:504] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.643122 20934 leader_election.cc:288] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cc5216405604ba6ae10ff1648736b05 (127.8.226.66:38151), 85e984f3ed484d8c92237e081930b7a1 (127.8.226.67:35887)
I20260925 15:21:01.650406 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.643458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:40515 (local address 127.8.226.66:38151)
0925 15:21:01.643714 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.643733 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.644297 (+ 564us) server_negotiation.cc:407] Connection header received
0925 15:21:01.644523 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.644545 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.644625 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.644713 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:01.645409 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.646348 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.648513 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.649477 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.649655 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.649726 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.650014 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.650128 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 15:21:01.650456 20631 tablet.cc:2369] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:01.650620 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.643478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:38151 (local address 127.8.226.65:40515)
0925 15:21:01.644190 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.644206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.644376 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.644808 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.644820 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.644841 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:01.645246 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.645258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.646483 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.646489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.648355 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.648370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.649678 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.649684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.649853 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.649879 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 15:21:01.649979 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:21:01.650104 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":78,"thread_start_us":411,"threads_started":1}
I20260925 15:21:01.650738 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.644394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:35887 (local address 127.8.226.65:46411)
0925 15:21:01.644891 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.644903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.645084 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.645399 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.645403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.645425 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:01.645803 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.645815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.646999 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.647003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.648478 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.648494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.649882 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.649888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.650033 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.650055 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:21:01.650163 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:21:01.650289 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":237,"threads_started":1}
I20260925 15:21:01.651651 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.644503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:46411 (local address 127.8.226.67:35887)
0925 15:21:01.644724 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.644729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.645018 (+ 289us) server_negotiation.cc:407] Connection header received
0925 15:21:01.645212 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.645217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.645288 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.645373 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:01.645997 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.646868 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.648649 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.649626 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.649813 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.649893 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.651289 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.651422 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:21:01.651774 20728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24ed6d87f3e4251b7bcace90b398d28" candidate_uuid: "e5992d9666dd4f8a83cec6c685895cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc5216405604ba6ae10ff1648736b05" is_pre_election: true
I20260925 15:21:01.652266 20728 raft_consensus.cc:2454] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5992d9666dd4f8a83cec6c685895cce in term 0.
I20260925 15:21:01.652915 20520 leader_election.cc:302] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc5216405604ba6ae10ff1648736b05, e5992d9666dd4f8a83cec6c685895cce; no voters:
I20260925 15:21:01.652913 20862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24ed6d87f3e4251b7bcace90b398d28" candidate_uuid: "e5992d9666dd4f8a83cec6c685895cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e984f3ed484d8c92237e081930b7a1" is_pre_election: true
I20260925 15:21:01.653322 20934 raft_consensus.cc:2781] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:21:01.653462 20934 raft_consensus.cc:482] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:21:01.653414 20862 raft_consensus.cc:2454] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5992d9666dd4f8a83cec6c685895cce in term 0.
I20260925 15:21:01.653559 20934 raft_consensus.cc:3037] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:01.655560 20934 raft_consensus.cc:504] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.656013 20934 leader_election.cc:288] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [CANDIDATE]: Term 1 election: Requested vote from peers 5cc5216405604ba6ae10ff1648736b05 (127.8.226.66:38151), 85e984f3ed484d8c92237e081930b7a1 (127.8.226.67:35887)
I20260925 15:21:01.656514 20728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24ed6d87f3e4251b7bcace90b398d28" candidate_uuid: "e5992d9666dd4f8a83cec6c685895cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc5216405604ba6ae10ff1648736b05"
I20260925 15:21:01.656570 20862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24ed6d87f3e4251b7bcace90b398d28" candidate_uuid: "e5992d9666dd4f8a83cec6c685895cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e984f3ed484d8c92237e081930b7a1"
I20260925 15:21:01.656701 20728 raft_consensus.cc:3037] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:01.656719 20862 raft_consensus.cc:3037] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:01.659816 20728 raft_consensus.cc:2454] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5992d9666dd4f8a83cec6c685895cce in term 1.
I20260925 15:21:01.659822 20862 raft_consensus.cc:2454] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5992d9666dd4f8a83cec6c685895cce in term 1.
I20260925 15:21:01.660158 20520 leader_election.cc:302] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e984f3ed484d8c92237e081930b7a1, e5992d9666dd4f8a83cec6c685895cce; no voters:
I20260925 15:21:01.660391 20934 raft_consensus.cc:2781] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:01.660807 20934 raft_consensus.cc:686] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 1 LEADER]: Becoming Leader. State: Replica: e5992d9666dd4f8a83cec6c685895cce, State: Running, Role: LEADER
I20260925 15:21:01.661229 20934 consensus_queue.cc:237] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } }
I20260925 15:21:01.665633 20417 catalog_manager.cc:5495] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce reported cstate change: term changed from 0 to 1, leader changed from <none> to e5992d9666dd4f8a83cec6c685895cce (127.8.226.65). New cstate: current_term: 1 leader_uuid: "e5992d9666dd4f8a83cec6c685895cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5992d9666dd4f8a83cec6c685895cce" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 36707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 } health_report { overall_health: UNKNOWN } } }
W20260925 15:21:01.680357 20908 tablet.cc:2369] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:01.696744 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.691583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.8.226.65:36707)
0925 15:21:01.691789 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.691795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.691846 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:21:01.692073 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.692076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.692147 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.692286 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:21:01.692662 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.693379 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.694754 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.695105 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.695233 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.695749 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:21:01.696202 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:21:01.696364 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.696419 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.696492 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:21:01.704115 20862 raft_consensus.cc:1261] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 1 FOLLOWER]: Refusing update from remote peer e5992d9666dd4f8a83cec6c685895cce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:01.704115 20728 raft_consensus.cc:1261] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 1 FOLLOWER]: Refusing update from remote peer e5992d9666dd4f8a83cec6c685895cce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:01.704994 20937 consensus_queue.cc:1035] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e984f3ed484d8c92237e081930b7a1" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:01.705322 20934 consensus_queue.cc:1035] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc5216405604ba6ae10ff1648736b05" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:01.727996 20416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:01.728276 20416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:01.731565 20416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa8f3abb3cf14c48b3ac9cbce72ad487: Sending DeleteTablet for 3 replicas of tablet d24ed6d87f3e4251b7bcace90b398d28
I20260925 15:21:01.732537 20708 tablet_service.cc:1514] Processing DeleteTablet for tablet d24ed6d87f3e4251b7bcace90b398d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:57836
I20260925 15:21:01.732645 20565 tablet_service.cc:1514] Processing DeleteTablet for tablet d24ed6d87f3e4251b7bcace90b398d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:33428
I20260925 15:21:01.733628 20842 tablet_service.cc:1514] Processing DeleteTablet for tablet d24ed6d87f3e4251b7bcace90b398d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:01 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:51734
I20260925 15:21:01.734635 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 20497
I20260925 15:21:01.745078 20951 tablet_replica.cc:330] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: stopping tablet replica
I20260925 15:21:01.745585 20951 raft_consensus.cc:2229] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:21:01.746344 20951 raft_consensus.cc:2258] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:01.751986 20951 ts_tablet_manager.cc:1905] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:01.753149 20950 tablet_replica.cc:330] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: stopping tablet replica
I20260925 15:21:01.753541 20950 raft_consensus.cc:2229] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:01.753773 20952 tablet_replica.cc:330] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: stopping tablet replica
I20260925 15:21:01.754024 20950 raft_consensus.cc:2258] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:01.754158 20952 raft_consensus.cc:2229] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:01.754637 20952 raft_consensus.cc:2258] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:01.757716 20950 ts_tablet_manager.cc:1905] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:21:01 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:21:01.759696 20951 ts_tablet_manager.cc:1918] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:01.759856 20951 log.cc:1196] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/d24ed6d87f3e4251b7bcace90b398d28
I20260925 15:21:01.760375 20951 ts_tablet_manager.cc:1939] T d24ed6d87f3e4251b7bcace90b398d28 P e5992d9666dd4f8a83cec6c685895cce: Deleting consensus metadata
I20260925 15:21:01.762233 20401 catalog_manager.cc:4841] TS e5992d9666dd4f8a83cec6c685895cce (127.8.226.65:36707): tablet d24ed6d87f3e4251b7bcace90b398d28 (table test-table [id=c6722355e16d49fc9fba2bf5217ac5d3]) successfully deleted
I20260925 15:21:01.763985 20950 ts_tablet_manager.cc:1918] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:01.764149 20950 log.cc:1196] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/d24ed6d87f3e4251b7bcace90b398d28
I20260925 15:21:01.764534 20950 ts_tablet_manager.cc:1939] T d24ed6d87f3e4251b7bcace90b398d28 P 5cc5216405604ba6ae10ff1648736b05: Deleting consensus metadata
I20260925 15:21:01.766039 20403 catalog_manager.cc:4841] TS 5cc5216405604ba6ae10ff1648736b05 (127.8.226.66:38151): tablet d24ed6d87f3e4251b7bcace90b398d28 (table test-table [id=c6722355e16d49fc9fba2bf5217ac5d3]) successfully deleted
I20260925 15:21:01.766525 20952 ts_tablet_manager.cc:1905] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:01.768539 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.735699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.8.226.65:36707)
0925 15:21:01.735903 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.735912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.746240 (+ 10328us) server_negotiation.cc:407] Connection header received
0925 15:21:01.746623 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.746630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.746728 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.747007 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.749283 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.750479 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.754507 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.755176 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.755289 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.761177 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:01.761191 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:01.761208 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:01.761252 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:01.765309 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.766122 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.766129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.766132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.766285 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.766723 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.766728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.766730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.767149 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:01.767413 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.767940 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.768180 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:21:01.770022 20625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:01.773793 20952 ts_tablet_manager.cc:1918] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:01.773969 20952 log.cc:1196] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/d24ed6d87f3e4251b7bcace90b398d28
I20260925 15:21:01.774408 20952 ts_tablet_manager.cc:1939] T d24ed6d87f3e4251b7bcace90b398d28 P 85e984f3ed484d8c92237e081930b7a1: Deleting consensus metadata
I20260925 15:21:01.776007 20403 catalog_manager.cc:4841] TS 85e984f3ed484d8c92237e081930b7a1 (127.8.226.67:35887): tablet d24ed6d87f3e4251b7bcace90b398d28 (table test-table [id=c6722355e16d49fc9fba2bf5217ac5d3]) successfully deleted
I20260925 15:21:01.865592 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 20497
I20260925 15:21:01.882671 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 20636
Sep 25 15:21:01 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:21:01.898396 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.883474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.8.226.66:38151)
0925 15:21:01.883639 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.883643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.883661 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:21:01.883766 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.883770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.883841 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.883923 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.885073 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.885728 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.886994 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.887649 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.887767 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.892121 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:01.892135 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:01.892152 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:01.892194 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:01.895772 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.896343 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.896352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.896356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.896511 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:01.896838 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:01.896842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:01.896846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:01.897325 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:01.897595 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:01.897775 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:21:01.897994 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:21:01.899044 20768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:01.985888 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 20636
I20260925 15:21:02.001511 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 20779
Sep 25 15:21:02 dist-test-slave-8vw7 krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:21:02.015085 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:02.002205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.8.226.67:35887)
0925 15:21:02.002369 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:02.002373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:02.002391 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:21:02.002566 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:02.002571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:02.002630 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:02.002710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:02.003916 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.004589 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.005822 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.006177 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.006273 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.010417 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:02.010431 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:02.010446 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:02.010480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:02.013024 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:02.013574 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:02.013580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:02.013582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:02.013691 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:02.013973 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:02.013975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:02.013977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:02.014304 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:02.014496 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:02.014708 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:21:02.014853 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:21:02.015657 20902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:02.099720 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 20779
I20260925 15:21:02.115396 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 20385
I20260925 15:21:02.116547 20447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:02.225009 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 20385
I20260925 15:21:02.248142 9097 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:21:02.583396 9097 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:21:02.583614 9097 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:21:02.929332 9097 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:21:02.929 UTC [19971] LOG: received smart shutdown request
2026-09-25 15:21:02.933 UTC [19971] LOG: background worker "logical replication launcher" (PID 19980) exited with exit code 1
2026-09-25 15:21:02.933 UTC [19975] LOG: shutting down
2026-09-25 15:21:03.083 UTC [19971] LOG: database system is shut down
2026-09-25T15:21:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45668 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:03 dist-test-slave-8vw7 krb5kdc[20966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:03 dist-test-slave-8vw7 krb5kdc[20966](info): set up 2 sockets
Sep 25 15:21:03 dist-test-slave-8vw7 krb5kdc[20966](info): commencing operation
krb5kdc: starting...
W20260925 15:21:04.257339 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:21:04 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/rangeradmin_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/rangeradmin_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/rangerlookup_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/rangerlookup_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
I20260925 15:21:04.555852 9097 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskN5mqX7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/postgres -l logfile start
2026-09-25 15:21:06.886 UTC [20996] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:21:06.886 UTC [20996] LOG: listening on IPv4 address "127.8.226.84", port 51877
2026-09-25 15:21:06.890 UTC [20996] LOG: listening on Unix socket "/tmp/.s.PGSQL.51877"
2026-09-25 15:21:06.910 UTC [20999] LOG: database system was shut down at 2026-09-25 15:21:05 UTC
2026-09-25 15:21:06.913 UTC [20996] LOG: database system is ready to accept connections
I20260925 15:21:07.841194 9097 mini_postgres.cc:96] Postgres bound to 51877
2026-09-25 15:21:07.857 UTC [21007] FATAL: database "slave" does not exist
127.8.226.84:51877 - accepting connections
I20260925 15:21:07.857896 9097 mini_ranger.cc:161] Starting Ranger...
I20260925 15:21:07.880857 9097 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:21:08.324056 9097 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:21:08.324174 9097 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-admin
2026-09-25 15:21:08,607 [I] DB FLAVOR :POSTGRES
2026-09-25 15:21:08,607 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:21:08,607 [I] Checking connection..
2026-09-25 15:21:08,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:21:08,843 [I] Checking connection passed.
2026-09-25 15:21:08,843 [I] --------- Verifying version history table ---------
2026-09-25 15:21:08,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:21:09,097 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:21:09,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:21:09,327 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:21:09,328 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:21:09,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:21:09.568 UTC [21071] WARNING: there is no transaction in progress
2026-09-25 15:21:09,574 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:21:09,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:21:09,792 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:21:09,792 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:21:09,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:21:10,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:21:10,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8vw7.c.gcp-upstream.internal','N') ;"
2026-09-25 15:21:10,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:21:10,725 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:21:10,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:21:10,974 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:21:10,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:21:11,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:21:11,445 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:21:11,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:21:12.323 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.338 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.352 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.366 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.381 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.400 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.457 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.463 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.476 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.485 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.495 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.507 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.516 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.522 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.529 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:12.535 UTC [21179] WARNING: there is no transaction in progress
2026-09-25 15:21:13,132 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:21:13,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:21:13,346 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:21:13,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:21:13,596 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:21:13,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:21:13,845 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:21:13,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:21:14,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:21:14,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8vw7.c.gcp-upstream.internal';"
2026-09-25 15:21:14,581 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:21:14,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:21:14,831 [I] DB_PATCHES have already been applied
I20260925 15:21:14.836935 9097 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-admin:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskN5mqX7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:21:14.837086 9097 mini_ranger.cc:193] Using host: 127.8.226.84
I20260925 15:21:14.851349 9097 mini_ranger.cc:227] Ranger admin URL: http://127.8.226.84:37597
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:21:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:21:15.914984 9097 mini_ranger.cc:160] Time spent starting Ranger: real 8.057s user 0.004s sys 0.030s
Sep 25, 2026 3:21:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:21:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 3:21:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:21:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:21:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:21:36.448024 9097 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.226.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.226.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/rangerkms_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.226.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/rangerkms_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.226.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.226.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.226.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/HTTP_127.8.226.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:21:36 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349696, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:21:36.619532 9097 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:21:36.636638 9097 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:21:37.446261 9097 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:21:37.446373 9097 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms
I20260925 15:21:37.446429 9097 mini_ranger_kms.cc:227] Using postgres at 127.8.226.84:51877
2026-09-25 15:21:37,780 [I] DB FLAVOR :POSTGRES
2026-09-25 15:21:37,780 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:21:37,780 [I] Checking connection
2026-09-25 15:21:37,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:21:38,002 [I] connection success
2026-09-25 15:21:38,003 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:21:38,003 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:21:38,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:21:38,233 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:21:38,233 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:21:38,233 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:21:38,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.226.84:51877/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:21:38,484 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:21:38.929037 9097 mini_ranger_kms.cc:310] Created kms service
I20260925 15:21:39.118686 9097 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:21:39.170073 9097 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:21:39 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349696, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:39.531215 9097 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:21:39.531500 9097 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms:/tmp/dist-test-taskN5mqX7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskN5mqX7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:21:39.531541 9097 mini_ranger_kms.cc:240] Using host: 127.8.226.84
I20260925 15:21:39.547876 9097 mini_ranger_kms.cc:276] Ranger KMS PID: 21359
I20260925 15:21:39.547984 9097 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.226.84:57689
Sep 25, 2026 3:21:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:21:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:21:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:21:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:21:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:21:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 15:21:41 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.226.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:41.801287 9097 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.182s user 0.061s sys 0.285s
I20260925 15:21:41.801545 9097 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:21:41.801757 9097 mini_ranger_kms.cc:401] 127.8.226.84:57689/kms/v1/keys
Sep 25 15:21:41 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.226.84@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
Sep 25 15:21:42 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.226.84@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
Sep 25 15:21:44 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:21:44.533962 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:37495
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:37495
--ranger_config_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:44.863938 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:44.864199 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:44.864284 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:44.871392 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:44.871490 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:44.871531 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:44.871580 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:44.871634 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:44.881193 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.226.84:57689/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:37495
--ranger_config_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:44.882328 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:44.883539 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:44.895810 21416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:44.895891 21415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:44.895870 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:44.896181 21410 server_base.cc:1034] running on GCE node
I20260925 15:21:44.897344 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:44.898828 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:44.900074 21410 hybrid_clock.cc:648] HybridClock initialized: now 1790349704900031 us; error 37 us; skew 500 ppm
I20260925 15:21:44.900416 21410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:44 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:44.908717 21410 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:21:44.910313 21410 webserver.cc:460] Webserver started at http://127.8.226.126:38089/ using document root <none> and password file <none>
I20260925 15:21:44.910981 21410 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:44.911118 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:44.911418 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:44 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:44.969010 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "c9e6d8006d824bb7bd893a48eb9769ff"
format_stamp: "Formatted at 2026-09-25 15:21:44 on dist-test-slave-8vw7"
server_key: "6f4418b2ed0c100e637e7bcc487b291c"
server_key_iv: "5911476ac6f205520cfb7390342897dc"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:44.970137 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "c9e6d8006d824bb7bd893a48eb9769ff"
format_stamp: "Formatted at 2026-09-25 15:21:44 on dist-test-slave-8vw7"
server_key: "6f4418b2ed0c100e637e7bcc487b291c"
server_key_iv: "5911476ac6f205520cfb7390342897dc"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:44.977278 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260925 15:21:44.997176 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:44.998883 21410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:21:44.999053 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "c9e6d8006d824bb7bd893a48eb9769ff"
format_stamp: "Formatted at 2026-09-25 15:21:44 on dist-test-slave-8vw7"
server_key: "6f4418b2ed0c100e637e7bcc487b291c"
server_key_iv: "5911476ac6f205520cfb7390342897dc"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:44.999279 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:45.020000 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:45.024684 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:45.025141 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:45.048280 21410 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:37495
I20260925 15:21:45.048326 21476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:37495 every 8 connection(s)
I20260925 15:21:45.049777 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:45.053968 21477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:45.058491 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 21410
I20260925 15:21:45.058723 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:21:45.059376 9097 external_mini_cluster.cc:1428] Setting key 456e3298c7263a24495451e662510336
I20260925 15:21:45.067167 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Bootstrap starting.
I20260925 15:21:45.071144 21477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:45.072499 21477 log.cc:824] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:45.075606 21477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: No bootstrap required, opened a new log
Sep 25 15:21:45 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:45.085203 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:45.061505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.8.226.126:37495)
0925 15:21:45.062485 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:45.062502 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:45.062563 (+ 61us) server_negotiation.cc:407] Connection header received
0925 15:21:45.065360 (+ 2797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:45.065394 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:45.065964 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:45.066467 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:45.067822 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:45.070174 (+ 2352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:45.071571 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:45.072732 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:45.072973 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:45.076913 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:45.076924 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:45.076943 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:45.077014 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:45.081025 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:45.081743 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:45.081752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:45.081764 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:45.081929 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:45.082334 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:45.082339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:45.082343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:45.082908 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:45.083233 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:45.083699 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 15:21:45.084125 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":367,"threads_started":1}
I20260925 15:21:45.085235 21477 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:45.085721 21477 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:45.085847 21477 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e6d8006d824bb7bd893a48eb9769ff, State: Initialized, Role: FOLLOWER
I20260925 15:21:45.086439 21477 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:45.086683 21477 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:45.086858 21477 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:45.087039 21477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:45.089730 21477 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:45.090287 21477 leader_election.cc:302] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9e6d8006d824bb7bd893a48eb9769ff; no voters:
I20260925 15:21:45.090938 21477 leader_election.cc:288] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:21:45.091051 21482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:45.092289 21482 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 LEADER]: Becoming Leader. State: Replica: c9e6d8006d824bb7bd893a48eb9769ff, State: Running, Role: LEADER
I20260925 15:21:45.093266 21477 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:45.093993 21482 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:45.096789 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e6d8006d824bb7bd893a48eb9769ff. Latest consensus state: current_term: 1 leader_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } } }
I20260925 15:21:45.096760 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } } }
I20260925 15:21:45.097334 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:45.097339 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:45.098352 21490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:45.105417 21490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:45.106984 21477 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:21:45.122030 21490 catalog_manager.cc:1347] Generated new cluster ID: 9333824f281a49bfb813605baa57b535
I20260925 15:21:45.122200 21490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:45.156097 21490 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:45.158259 21490 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:45.168020 21490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Generated new TSK 0
I20260925 15:21:45.169308 21490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:21:45.810523 21519 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:21:45.902230 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37495
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:21:46 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:21:46.323444 21522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:46.323909 21522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:46.324100 21522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:46.336087 21522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:46.336242 21522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:46.336494 21522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:21:46.352355 21522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.226.84:57689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:46.353971 21522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:46.355645 21522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:46.372164 21532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:46.373895 21533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:46.382570 21535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:46.383153 21522 server_base.cc:1034] running on GCE node
I20260925 15:21:46.383934 21522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:46.400110 21522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:46.401526 21522 hybrid_clock.cc:648] HybridClock initialized: now 1790349706401406 us; error 86 us; skew 500 ppm
I20260925 15:21:46.402006 21522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:46 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:46.406345 21522 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:21:46.408974 21522 webserver.cc:460] Webserver started at http://127.8.226.65:38095/ using document root <none> and password file <none>
I20260925 15:21:46.410002 21522 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:46.410156 21522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:46.410548 21522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:46 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:46.430325 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "0ffb800fc4f6489cbe9130a39125d7d1"
format_stamp: "Formatted at 2026-09-25 15:21:46 on dist-test-slave-8vw7"
server_key: "e78243099b8637aa7e6944e0b8d3f828"
server_key_iv: "6f75befcae95435799ed705fb87d966f"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:46.431434 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "0ffb800fc4f6489cbe9130a39125d7d1"
format_stamp: "Formatted at 2026-09-25 15:21:46 on dist-test-slave-8vw7"
server_key: "e78243099b8637aa7e6944e0b8d3f828"
server_key_iv: "6f75befcae95435799ed705fb87d966f"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:46.438555 21522 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 15:21:46.481676 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:46.483783 21522 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.008s sys 0.008s
I20260925 15:21:46.484000 21522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "0ffb800fc4f6489cbe9130a39125d7d1"
format_stamp: "Formatted at 2026-09-25 15:21:46 on dist-test-slave-8vw7"
server_key: "e78243099b8637aa7e6944e0b8d3f828"
server_key_iv: "6f75befcae95435799ed705fb87d966f"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:46.484236 21522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:46.527597 21522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:46.533077 21522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:46.533481 21522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:46.536358 21522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:46.541826 21522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:46.541986 21522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:46.542182 21522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:46.542301 21522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:21:46 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:46.697185 21522 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:39801
I20260925 15:21:46.697201 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:39801 every 8 connection(s)
I20260925 15:21:46.698580 21522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:46.698951 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 21522
I20260925 15:21:46.699167 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:21:46.700170 9097 external_mini_cluster.cc:1428] Setting key cda86923b1ac1d8054436eca92f9d202
Sep 25 15:21:46 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
I20260925 15:21:46.739856 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:46.709153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:52399 (local address 127.8.226.126:37495)
0925 15:21:46.718785 (+ 9632us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:46.718794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:46.719529 (+ 735us) server_negotiation.cc:407] Connection header received
0925 15:21:46.721771 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:46.721777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:46.721880 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:46.722002 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:46.724023 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:46.725156 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:46.727832 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:46.728430 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:46.728567 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:46.733179 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:46.733184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:46.733188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:46.733223 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:46.736603 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:46.737384 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:46.737421 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:46.737425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:46.737563 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:46.737990 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:46.737995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:46.737998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:46.738336 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:46.738506 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:46.739231 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 15:21:46.739448 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9462,"thread_start_us":11169,"threads_started":1}
I20260925 15:21:46.741180 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:46.708389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37495 (local address 127.8.226.65:52399)
0925 15:21:46.719335 (+ 10946us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:46.719401 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:46.721469 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:46.722305 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:46.722319 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:46.722976 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:46.723811 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:46.723837 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:46.725324 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:46.725333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:46.727646 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:46.727666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:46.728684 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:46.728694 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:46.729060 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:46.729926 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:46.729966 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:46.732882 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:46.736760 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:46.736767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:46.736781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:46.737271 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:46.737690 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:46.737693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:46.737696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:46.737875 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:46.738451 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:46.738457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:46.738856 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 15:21:46.739106 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 15:21:46.739681 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10572,"thread_start_us":10938,"threads_started":1}
I20260925 15:21:46.742973 21655 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37495
I20260925 15:21:46.743315 21655 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:46.744138 21655 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:46.746614 21442 ts_manager.cc:194] Registered new tserver with Master: 0ffb800fc4f6489cbe9130a39125d7d1 (127.8.226.65:39801)
I20260925 15:21:46.749455 21442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:52399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:21:46.774629 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37495
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:21:46 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
W20260925 15:21:47.236119 21661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:47.236585 21661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:47.236791 21661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:47.248592 21661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:47.248828 21661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:47.249107 21661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:21:47.265206 21661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.226.84:57689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:47.267269 21661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:47.269281 21661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:47.287035 21675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:47.287708 21676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:47.302655 21678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:47.303251 21661 server_base.cc:1034] running on GCE node
I20260925 15:21:47.304023 21661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:47.305145 21661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:47.306380 21661 hybrid_clock.cc:648] HybridClock initialized: now 1790349707306322 us; error 53 us; skew 500 ppm
I20260925 15:21:47.306722 21661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:47 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349707, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:47.310043 21661 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:21:47.311695 21661 webserver.cc:460] Webserver started at http://127.8.226.66:39623/ using document root <none> and password file <none>
I20260925 15:21:47.312438 21661 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:47.312567 21661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:47.312887 21661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:47 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349707, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:47.330968 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "5262a57831434f78bad321ef8f5923c8"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "380641f114ce7d7e9a03baafce76c322"
server_key_iv: "d2f4b33bc79e3054f004088c48451556"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:47.332031 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "5262a57831434f78bad321ef8f5923c8"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "380641f114ce7d7e9a03baafce76c322"
server_key_iv: "d2f4b33bc79e3054f004088c48451556"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:47.339205 21661 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 15:21:47.357350 21685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:47.359316 21661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:21:47.359586 21661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "5262a57831434f78bad321ef8f5923c8"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "380641f114ce7d7e9a03baafce76c322"
server_key_iv: "d2f4b33bc79e3054f004088c48451556"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:47.359942 21661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:47.388038 21661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:47.393464 21661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:47.393852 21661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:47.395092 21661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:47.396885 21661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:47.397042 21661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:21:47.397226 21661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:47.397322 21661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:47.435958 21661 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:34209
I20260925 15:21:47.435958 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:34209 every 8 connection(s)
I20260925 15:21:47.437456 21661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:21:47.444772 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 21661
I20260925 15:21:47.444933 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:21:47.445506 9097 external_mini_cluster.cc:1428] Setting key 122c6bdb3ee45754b0299085e45ce908
Sep 25 15:21:47 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349707, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:47.461452 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:47.440240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:44859 (local address 127.8.226.126:37495)
0925 15:21:47.440714 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:47.440718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:47.441857 (+ 1139us) server_negotiation.cc:407] Connection header received
0925 15:21:47.443645 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:47.443651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:47.443756 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:47.443879 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:47.445575 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.446491 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.448552 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.449171 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.449305 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:47.453980 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:47.453989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:47.453998 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:47.454047 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:47.457278 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:47.458296 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:47.458304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:47.458311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:47.458472 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:47.459039 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:47.459047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:47.459053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:47.459470 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:47.459628 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:47.460526 (+ 898us) server_negotiation.cc:299] Negotiation successful
0925 15:21:47.460897 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":260,"threads_started":1}
I20260925 15:21:47.462872 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:47.440882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37495 (local address 127.8.226.66:44859)
0925 15:21:47.441669 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:47.441732 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:47.443374 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:47.444121 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:47.444131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:47.444570 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:47.445388 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:47.445414 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.446651 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.446657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:47.448380 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:47.448393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.449432 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.449442 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:47.449814 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:47.450621 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:47.450656 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:47.453680 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:47.457452 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:47.457464 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:47.457484 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:47.458116 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:47.458608 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:47.458613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:47.458617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:47.458896 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:47.459621 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:47.459630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:47.460074 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 15:21:47.460380 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 15:21:47.461068 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":304,"threads_started":1}
I20260925 15:21:47.464592 21798 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37495
I20260925 15:21:47.464936 21798 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:47.465890 21798 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:47.467784 21442 ts_manager.cc:194] Registered new tserver with Master: 5262a57831434f78bad321ef8f5923c8 (127.8.226.66:34209)
I20260925 15:21:47.468854 21442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:44859
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:21:47.505805 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37495
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:21:47.754355 21655 heartbeater.cc:503] Master 127.8.226.126:37495 was elected leader, sending a full tablet report...
W20260925 15:21:47.842029 21804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:47.842332 21804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:47.842507 21804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:47.850034 21804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:47.850171 21804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:47.850350 21804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:21:47.860474 21804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.226.84:57689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:47.861920 21804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:47.863304 21804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:47.876928 21809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:47.877056 21810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:47.877080 21812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:47.877940 21804 server_base.cc:1034] running on GCE node
I20260925 15:21:47.878551 21804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:47.879530 21804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:47.880750 21804 hybrid_clock.cc:648] HybridClock initialized: now 1790349707880677 us; error 72 us; skew 500 ppm
I20260925 15:21:47.881114 21804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:47 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349707, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:47.884142 21804 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:21:47.885603 21804 webserver.cc:460] Webserver started at http://127.8.226.67:41981/ using document root <none> and password file <none>
I20260925 15:21:47.886312 21804 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:47.886456 21804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:47.886766 21804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:47 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349707, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:47.903156 21804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "a379b95789214efe9d513b320872b0c3"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "78f7499784283d43123fc2ca58b9a7a2"
server_key_iv: "c8cc0e10062d3ff6b64a93a8f4c35338"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:47.904139 21804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "a379b95789214efe9d513b320872b0c3"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "78f7499784283d43123fc2ca58b9a7a2"
server_key_iv: "c8cc0e10062d3ff6b64a93a8f4c35338"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:47.910082 21804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 15:21:47.927000 21819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:47.928711 21804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 15:21:47.928886 21804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "a379b95789214efe9d513b320872b0c3"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "78f7499784283d43123fc2ca58b9a7a2"
server_key_iv: "c8cc0e10062d3ff6b64a93a8f4c35338"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:47.929195 21804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:47.945845 21804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:47.949977 21804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:47.950322 21804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:47.951262 21804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:47.952553 21804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:47.952625 21804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:47.952773 21804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:47.952822 21804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:47.986014 21804 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:43273
I20260925 15:21:47.986045 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:43273 every 8 connection(s)
I20260925 15:21:47.987404 21804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:47.989744 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 21804
I20260925 15:21:47.989961 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:21:47.990583 9097 external_mini_cluster.cc:1428] Setting key 52dd63bdae0217693815e8e072938d88
Sep 25 15:21:47 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349707, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:48.008456 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:47.990219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:43237 (local address 127.8.226.126:37495)
0925 15:21:47.990708 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:47.990716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:47.991950 (+ 1234us) server_negotiation.cc:407] Connection header received
0925 15:21:47.994067 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:47.994072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:47.994135 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:47.994219 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:47.996093 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.996825 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.998378 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.998799 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.998901 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:48.002251 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:48.002256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:48.002259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:48.002303 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:48.005763 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.006542 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.006547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.006549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.006671 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.007048 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.007051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.007053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.007323 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:48.007430 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:48.008024 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 15:21:48.008180 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":252,"threads_started":1}
I20260925 15:21:48.009455 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:47.990882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37495 (local address 127.8.226.67:43237)
0925 15:21:47.991748 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:47.991804 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:47.993815 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:47.994467 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:47.994475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:47.994971 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:47.995905 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:47.995931 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.997005 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.997010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:47.998236 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:47.998247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:47.998984 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:47.998989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:47.999201 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:47.999821 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:47.999842 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:48.002002 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:48.005899 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:48.005905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:48.005955 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:48.006428 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:48.006762 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:48.006765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:48.006767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:48.006950 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:48.007429 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:48.007435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:48.007723 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 15:21:48.007939 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 15:21:48.008315 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":395,"threads_started":1}
I20260925 15:21:48.010758 21932 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37495
I20260925 15:21:48.011044 21932 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:48.011709 21932 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:48.013106 21442 ts_manager.cc:194] Registered new tserver with Master: a379b95789214efe9d513b320872b0c3 (127.8.226.67:43273)
I20260925 15:21:48.013918 21442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:43237
I20260925 15:21:48.018312 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:48.019975 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 21522
Sep 25 15:21:48 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:21:48.036031 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:48.020836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.8.226.65:39801)
0925 15:21:48.021610 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:48.021616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:48.021639 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:21:48.021693 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:48.021698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:48.021895 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:48.022010 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:48.023478 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:48.024468 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:48.025996 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:48.026506 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:48.026599 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:48.030670 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:48.030681 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:48.030697 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:48.030732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:48.033535 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.034249 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.034256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.034258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.034390 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.034690 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.034694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.034697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.035010 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:48.035241 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:48.035460 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:21:48.035753 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":472,"threads_started":1}
I20260925 15:21:48.037312 21650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:48.121999 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 21522
I20260925 15:21:48.138882 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 21661
Sep 25 15:21:48 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:21:48.154867 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:48.139851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.8.226.66:34209)
0925 15:21:48.140408 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:48.140416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:48.140440 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:21:48.140503 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:48.140509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:48.140733 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:48.140865 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:48.142107 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:48.143210 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:48.144551 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:48.145142 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:48.145245 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:48.149423 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:48.149434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:48.149448 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:48.149480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:48.152309 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.152876 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.152883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.152885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.153074 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.153435 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.153439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.153441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.153761 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:48.153984 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:48.154282 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:21:48.154483 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
I20260925 15:21:48.156103 21793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:48.240204 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 21661
I20260925 15:21:48.255545 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 21804
Sep 25 15:21:48 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:21:48.272929 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:48.256554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.8.226.67:43273)
0925 15:21:48.257233 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:48.257241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:48.257272 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:21:48.257329 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:48.257335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:48.257549 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:48.257717 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:48.259037 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:48.260021 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:48.261453 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:48.262022 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:48.262163 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:48.266932 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:48.266945 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:48.266961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:48.267004 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:48.270232 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.270903 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.270911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.270913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.271049 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:48.271448 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:48.271453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:48.271456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:48.271867 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:48.272121 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:48.272384 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:21:48.272598 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":316,"threads_started":1}
I20260925 15:21:48.274300 21927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:48.359592 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 21804
I20260925 15:21:48.375495 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 21410
I20260925 15:21:48.377063 21472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:48.487695 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 21410
I20260925 15:21:48.511000 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:37495
--webserver_interface=127.8.226.126
--webserver_port=38089
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:37495
--ranger_config_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:48.835698 21944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:48.836062 21944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:48.836241 21944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:48.843582 21944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:48.843736 21944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:48.843838 21944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:48.843940 21944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:48.844019 21944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:48.854039 21944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:37495
--ranger_config_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:48.855350 21944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:48.856673 21944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:48.868204 21949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:48.868250 21950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:48.868158 21952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:48.869364 21944 server_base.cc:1034] running on GCE node
I20260925 15:21:48.870119 21944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:48.871477 21944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:48.872705 21944 hybrid_clock.cc:648] HybridClock initialized: now 1790349708872650 us; error 49 us; skew 500 ppm
I20260925 15:21:48.873139 21944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:48 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349708, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:48.876049 21944 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:21:48.877559 21944 webserver.cc:460] Webserver started at http://127.8.226.126:38089/ using document root <none> and password file <none>
I20260925 15:21:48.878201 21944 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:48.878355 21944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:21:48 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349708, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:48.954162 21944 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:21:48.957223 21960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:48.958560 21944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:21:48.958765 21944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "c9e6d8006d824bb7bd893a48eb9769ff"
format_stamp: "Formatted at 2026-09-25 15:21:44 on dist-test-slave-8vw7"
server_key: "6f4418b2ed0c100e637e7bcc487b291c"
server_key_iv: "5911476ac6f205520cfb7390342897dc"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:48.959713 21944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:48.972311 21944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:48.976372 21944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:48.976651 21944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:48.995172 21944 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:37495
I20260925 15:21:48.995194 22011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:37495 every 8 connection(s)
I20260925 15:21:48.996642 21944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:49.003190 22012 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:21:49.005504 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 21944
I20260925 15:21:49.006381 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:39801
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:37495
--webserver_port=38095
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:21:49.006556 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Bootstrap starting.
I20260925 15:21:49.025573 22012 log.cc:824] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:49.034858 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:21:49.035573 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Bootstrap complete.
I20260925 15:21:49.049398 22012 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:49.050331 22012 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e6d8006d824bb7bd893a48eb9769ff, State: Initialized, Role: FOLLOWER
I20260925 15:21:49.051061 22012 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:49.051307 22012 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:49.051436 22012 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:49.051638 22012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:21:49.055806 22012 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:49.056317 22012 leader_election.cc:302] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9e6d8006d824bb7bd893a48eb9769ff; no voters:
I20260925 15:21:49.057017 22012 leader_election.cc:288] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:21:49.057216 22016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:21:49.058352 22016 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [term 2 LEADER]: Becoming Leader. State: Replica: c9e6d8006d824bb7bd893a48eb9769ff, State: Running, Role: LEADER
I20260925 15:21:49.059001 22016 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } }
I20260925 15:21:49.059335 22012 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:49.061965 22018 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e6d8006d824bb7bd893a48eb9769ff. Latest consensus state: current_term: 2 leader_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } } }
I20260925 15:21:49.062484 22018 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:49.063119 22017 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e6d8006d824bb7bd893a48eb9769ff" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37495 } } }
I20260925 15:21:49.063491 22017 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:49.066220 22021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:49.071815 22021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:49.073280 22021 catalog_manager.cc:1259] Loaded cluster ID: 9333824f281a49bfb813605baa57b535
I20260925 15:21:49.073417 22021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:49.077318 22021 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:49.078756 22021 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Loaded TSK: 0
I20260925 15:21:49.080322 22021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:21:49.087437 22012 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 15:21:49.489121 22014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:49.489712 22014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:49.489943 22014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:49.503052 22014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:49.503206 22014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:49.503481 22014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:21:49.521373 22014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:39801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=38095
--tserver_master_addrs=127.8.226.126:37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:49.523399 22014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:49.525338 22014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:49.545240 22049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:49.547484 22050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:49.557739 22052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:49.558924 22014 server_base.cc:1034] running on GCE node
I20260925 15:21:49.559865 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:49.561221 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:49.562539 22014 hybrid_clock.cc:648] HybridClock initialized: now 1790349709562470 us; error 60 us; skew 500 ppm
I20260925 15:21:49.563038 22014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:49 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349709, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:49.567838 22014 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:21:49.593393 22014 webserver.cc:460] Webserver started at http://127.8.226.65:38095/ using document root <none> and password file <none>
I20260925 15:21:49.594604 22014 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:49.594864 22014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:21:49 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349709, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:49.636755 22014 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 15:21:49.640662 22060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:49.643666 22014 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260925 15:21:49.643846 22014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "0ffb800fc4f6489cbe9130a39125d7d1"
format_stamp: "Formatted at 2026-09-25 15:21:46 on dist-test-slave-8vw7"
server_key: "e78243099b8637aa7e6944e0b8d3f828"
server_key_iv: "6f75befcae95435799ed705fb87d966f"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:49.645313 22014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:49.681715 22014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:49.687006 22014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:49.687343 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:49.690188 22014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:49.691792 22014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:49.691962 22014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:49.692138 22014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:49.692296 22014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:49.732463 22014 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:39801
I20260925 15:21:49.732534 22173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:39801 every 8 connection(s)
I20260925 15:21:49.734297 22014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:49.740942 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22014
I20260925 15:21:49.741780 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:34209
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:37495
--webserver_port=39623
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:21:49 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349709, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:49.774256 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:49.738042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37495 (local address 127.8.226.65:42935)
0925 15:21:49.739820 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:49.739871 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:49.748628 (+ 8757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:49.752139 (+ 3511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:49.752152 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:49.752836 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:49.753634 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:49.753670 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:49.757043 (+ 3373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:49.757052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:49.759706 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:49.759721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:49.761397 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:49.761405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:49.761675 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:49.762585 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:49.762617 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:49.765565 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:49.769886 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:49.769895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:49.769909 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:49.770328 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:49.770774 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:49.770779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:49.770782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:49.770970 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:49.771669 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:49.771677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:49.772089 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 15:21:49.772350 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:21:49.772899 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"thread_start_us":1286,"threads_started":1}
I20260925 15:21:49.774256 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:49.737784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:42935 (local address 127.8.226.126:37495)
0925 15:21:49.739209 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:49.739234 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:49.740045 (+ 811us) server_negotiation.cc:407] Connection header received
0925 15:21:49.749408 (+ 9363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:49.749434 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:49.749960 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:49.750399 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:49.753895 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:49.755940 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:49.759859 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:49.761137 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:49.761332 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:49.765881 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:49.765893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:49.765913 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:49.765969 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:49.769737 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:49.770474 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:49.770482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:49.770490 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:49.770652 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:49.771087 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:49.771092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:49.771095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:49.771528 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:49.771732 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:49.772574 (+ 842us) server_negotiation.cc:299] Negotiation successful
0925 15:21:49.772944 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":275,"threads_started":1}
I20260925 15:21:49.777887 22174 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37495
I20260925 15:21:49.778205 22174 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:49.779342 22174 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
W20260925 15:21:49.782512 22174 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:37495 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:21:49.782747 22174 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:49.783002 22174 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:49.784066 22174 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:49.784330 22174 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
W20260925 15:21:49.785370 22174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:21:50.160873 22178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:50.161233 22178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:50.161391 22178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:50.169557 22178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:50.169726 22178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:50.169898 22178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:21:50.179824 22178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:34209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=39623
--tserver_master_addrs=127.8.226.126:37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:50.181681 22178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:50.183384 22178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:50.199173 22192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:50.200237 22193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:50.201663 22195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:50.201742 22178 server_base.cc:1034] running on GCE node
I20260925 15:21:50.202575 22178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:50.203890 22178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:50.205175 22178 hybrid_clock.cc:648] HybridClock initialized: now 1790349710205105 us; error 49 us; skew 500 ppm
I20260925 15:21:50.205574 22178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:50.214512 22178 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:21:50.217324 22178 webserver.cc:460] Webserver started at http://127.8.226.66:39623/ using document root <none> and password file <none>
I20260925 15:21:50.218065 22178 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:50.218233 22178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:50.249691 22178 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.005s
I20260925 15:21:50.259734 22203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:50.261929 22178 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.003s
I20260925 15:21:50.262194 22178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "5262a57831434f78bad321ef8f5923c8"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "380641f114ce7d7e9a03baafce76c322"
server_key_iv: "d2f4b33bc79e3054f004088c48451556"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:50.263796 22178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:50.313547 22178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:50.317600 22178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:50.317915 22178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:50.318992 22178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:50.320462 22178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:50.320580 22178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:50.320720 22178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:50.320807 22178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:50.362675 22178 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:34209
I20260925 15:21:50.362730 22317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:34209 every 8 connection(s)
I20260925 15:21:50.364413 22178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:50.373948 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22178
I20260925 15:21:50.374773 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.226.84:57689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:43273
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:37495
--webserver_port=41981
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:50.396806 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:50.368491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37495 (local address 127.8.226.66:57621)
0925 15:21:50.371215 (+ 2724us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:50.371274 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:50.375549 (+ 4275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:50.377582 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:50.377591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:50.378024 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:50.378609 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:50.378625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:50.380284 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:50.380290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:50.382487 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:50.382504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:50.383815 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:50.383823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:50.384151 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:50.384939 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:50.384993 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:50.387758 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:50.391906 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:50.391915 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:50.391937 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:50.392536 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:50.393006 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:50.393010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:50.393014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:50.393270 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:50.394025 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:50.394034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:50.394464 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 15:21:50.394820 (+ 356us) client_negotiation.cc:240] Negotiation successful
0925 15:21:50.395407 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2372,"thread_start_us":1088,"threads_started":1}
I20260925 15:21:50.397930 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:50.369255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:57621 (local address 127.8.226.126:37495)
0925 15:21:50.376622 (+ 7367us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:50.376630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:50.376654 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:21:50.376733 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:50.376738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:50.376818 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:50.376936 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:50.378801 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:50.380075 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:50.382682 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:50.383362 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:50.383485 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:50.388035 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:50.388043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:50.388048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:50.388088 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:50.391735 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:50.392689 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:50.392695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:50.392698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:50.392856 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:50.393415 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:50.393422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:50.393427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:50.393800 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:50.393956 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:50.397339 (+ 3383us) server_negotiation.cc:299] Negotiation successful
0925 15:21:50.397573 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7216,"thread_start_us":7089,"threads_started":1}
I20260925 15:21:50.400198 22318 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37495
I20260925 15:21:50.400516 22318 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:50.401418 22318 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:50.403498 21977 ts_manager.cc:194] Registered new tserver with Master: 5262a57831434f78bad321ef8f5923c8 (127.8.226.66:34209)
I20260925 15:21:50.405833 21977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:57621
W20260925 15:21:50.770467 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:50.770862 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:50.771018 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:50.782732 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:50.782879 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:50.783072 22322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:21:50.785529 22174 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:50.785940 22174 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:50.787088 21977 ts_manager.cc:194] Registered new tserver with Master: 0ffb800fc4f6489cbe9130a39125d7d1 (127.8.226.65:39801)
I20260925 15:21:50.788203 21977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:42935
I20260925 15:21:50.798885 22322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.226.84:57689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:43273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=41981
--tserver_master_addrs=127.8.226.126:37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:50.800520 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:50.802287 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:50.827165 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:50.827687 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:50.828418 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:50.834882 22322 server_base.cc:1034] running on GCE node
I20260925 15:21:50.835750 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:50.838080 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:50.839335 22322 hybrid_clock.cc:648] HybridClock initialized: now 1790349710839220 us; error 75 us; skew 500 ppm
I20260925 15:21:50.839701 22322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:50.843586 22322 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:21:50.866154 22322 webserver.cc:460] Webserver started at http://127.8.226.67:41981/ using document root <none> and password file <none>
I20260925 15:21:50.867163 22322 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:50.867336 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:21:50 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:50.921736 22322 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.009s sys 0.000s
I20260925 15:21:50.932181 22339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:50.934548 22322 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.010s sys 0.000s
I20260925 15:21:50.934808 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "a379b95789214efe9d513b320872b0c3"
format_stamp: "Formatted at 2026-09-25 15:21:47 on dist-test-slave-8vw7"
server_key: "78f7499784283d43123fc2ca58b9a7a2"
server_key_iv: "c8cc0e10062d3ff6b64a93a8f4c35338"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:50.936393 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:50.962328 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:50.966358 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:50.966642 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:50.967774 22322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:50.969401 22322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:50.969517 22322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:50.969657 22322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:50.969746 22322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:51.048853 22322 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:43273
I20260925 15:21:51.048923 22451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:43273 every 8 connection(s)
I20260925 15:21:51.050925 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:51.053617 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22322
Sep 25 15:21:51 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:51.106184 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.056020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.8.226.126:37495)
0925 15:21:51.081377 (+ 25357us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.081383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.081403 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:21:51.081467 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.081472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.081557 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.081727 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:51.085093 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.086288 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.088377 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.089093 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.089215 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.093158 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:51.093166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:51.093171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:51.093219 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:51.101418 (+ 8199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.103020 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.103026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.103029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.103219 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.103696 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.103702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.103705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.104071 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:51.104330 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.105543 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.105760 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":25233,"thread_start_us":11160,"threads_started":1}
I20260925 15:21:51.107052 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.055717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:50359 (local address 127.8.226.126:37495)
0925 15:21:51.067945 (+ 12228us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.067956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.082711 (+ 14755us) server_negotiation.cc:407] Connection header received
0925 15:21:51.085171 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.085176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.085260 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.085350 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:51.087687 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.088858 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.091398 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.092173 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.092279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.098600 (+ 6321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:51.098611 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:51.098617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:51.098690 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:51.103049 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.103978 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.103983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.103987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.104129 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.104697 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.104702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.104706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.105090 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:51.105224 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.106205 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.106694 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":11605,"thread_start_us":25383,"threads_started":1}
I20260925 15:21:51.109509 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.054813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37495 (local address 127.8.226.67:50359)
0925 15:21:51.082508 (+ 27695us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.082577 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:51.084794 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:51.085951 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:51.085964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:51.086632 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:51.087457 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.087480 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.089073 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.089078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.091216 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.091233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.092651 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.092659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.093021 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.094433 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:51.094470 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:51.098263 (+ 3793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:51.103234 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:51.103244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:51.103263 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:51.103842 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:51.104298 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:51.104302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:51.104306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:51.104564 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:51.105315 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:51.105323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:51.105771 (+ 448us) client_negotiation.cc:769] Sending connection context
0925 15:21:51.107537 (+ 1766us) client_negotiation.cc:240] Negotiation successful
0925 15:21:51.108096 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27330,"thread_start_us":31001,"threads_started":1}
I20260925 15:21:51.111099 22452 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37495
I20260925 15:21:51.111418 22452 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:51.112267 22452 heartbeater.cc:511] Master 127.8.226.126:37495 requested a full tablet report, sending...
I20260925 15:21:51.113903 21977 ts_manager.cc:194] Registered new tserver with Master: a379b95789214efe9d513b320872b0c3 (127.8.226.67:43273)
I20260925 15:21:51.114993 21977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:50359
I20260925 15:21:51.123055 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:51.170186 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.145509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.8.226.126:37495)
0925 15:21:51.145739 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.145745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.152552 (+ 6807us) server_negotiation.cc:407] Connection header received
0925 15:21:51.152870 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.152876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.153003 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.153107 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:51.154728 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.155793 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.157665 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.158326 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.158431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.161869 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:51.161876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:51.161882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:51.161919 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:51.165439 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.167560 (+ 2121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.167566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.167570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.167738 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.168201 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.168206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.168210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.168587 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:51.168860 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.169482 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.169793 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 15:21:51 dist-test-slave-8vw7 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349710, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for HTTP/127.8.226.84@KRBTEST.COM
I20260925 15:21:51.175094 21977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:21:51.186815 21977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:21:51.229276 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.210669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:34209 (local address 127.0.0.1:42316)
0925 15:21:51.211905 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.211948 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:51.212107 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:51.213929 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:51.213941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:51.213991 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:51.214371 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.214390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.217989 (+ 3599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.217998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.222237 (+ 4239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.222256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.224660 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.224667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.224888 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.224942 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:21:51.225139 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 15:21:51.228843 (+ 3704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":411,"threads_started":1}
I20260925 15:21:51.242637 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.210766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.8.226.66:34209)
0925 15:21:51.211503 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.211513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.212297 (+ 784us) server_negotiation.cc:407] Connection header received
0925 15:21:51.212404 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.212412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.212675 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.213571 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:51.214646 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.217786 (+ 3140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.222451 (+ 4665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.224412 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.224540 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.224812 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.242010 (+ 17198us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.242320 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":458,"threads_started":1}
I20260925 15:21:51.247041 22253 tablet_service.cc:1467] Processing CreateTablet for tablet 795409c15243459c9ceeb0aab5a8e8a1 (DEFAULT_TABLE table=test-table [id=22650d9f0df84a83b3e885ef94dcb779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:51.248754 22253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795409c15243459c9ceeb0aab5a8e8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:51.248885 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.225889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42366)
0925 15:21:51.229551 (+ 3662us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.229571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:51.229701 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:51.234418 (+ 4717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:51.234425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:51.234456 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:51.234662 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.234675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.239968 (+ 5293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.239973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.244775 (+ 4802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.244789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.248127 (+ 3338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.248151 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.248334 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.248368 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:21:51.248441 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:21:51.248585 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3551,"thread_start_us":3277,"threads_started":1}
I20260925 15:21:51.249670 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.226507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.8.226.65:39801)
0925 15:21:51.227167 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.227179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.229701 (+ 2522us) server_negotiation.cc:407] Connection header received
0925 15:21:51.229877 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.229890 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.230247 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.230485 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:51.234895 (+ 4410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.236848 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.244978 (+ 8130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.246499 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.246614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.247975 (+ 1361us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.249181 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.249353 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":319,"threads_started":1}
I20260925 15:21:51.253772 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.212363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:43273 (local address 127.0.0.1:57756)
0925 15:21:51.230816 (+ 18453us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.230834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:51.231014 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:51.233666 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:51.233674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:51.233709 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:51.234204 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.234219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.241079 (+ 6860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.241085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.243995 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.244011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.252979 (+ 8968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.252986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.253164 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.253197 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:21:51.253294 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:21:51.253425 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16959,"thread_start_us":39508,"threads_started":1}
I20260925 15:21:51.253319 22109 tablet_service.cc:1467] Processing CreateTablet for tablet 795409c15243459c9ceeb0aab5a8e8a1 (DEFAULT_TABLE table=test-table [id=22650d9f0df84a83b3e885ef94dcb779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:51.253912 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.212979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.8.226.67:43273)
0925 15:21:51.226438 (+ 13459us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.226448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.230919 (+ 4471us) server_negotiation.cc:407] Connection header received
0925 15:21:51.232310 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.232323 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.232682 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.232896 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:51.237084 (+ 4188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.239782 (+ 2698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.250805 (+ 11023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.252319 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.252429 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.252688 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.253402 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.253579 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12209,"thread_start_us":12045,"threads_started":1}
I20260925 15:21:51.254818 22109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795409c15243459c9ceeb0aab5a8e8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:51.257371 22387 tablet_service.cc:1467] Processing CreateTablet for tablet 795409c15243459c9ceeb0aab5a8e8a1 (DEFAULT_TABLE table=test-table [id=22650d9f0df84a83b3e885ef94dcb779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:51.258615 22387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795409c15243459c9ceeb0aab5a8e8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:51.279050 22474 tablet_bootstrap.cc:492] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Bootstrap starting.
I20260925 15:21:51.279832 22475 tablet_bootstrap.cc:492] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Bootstrap starting.
I20260925 15:21:51.283042 22474 tablet_bootstrap.cc:654] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:51.284421 22474 log.cc:824] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:51.284578 22475 tablet_bootstrap.cc:654] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:51.285862 22475 log.cc:824] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:51.293754 22474 tablet_bootstrap.cc:492] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: No bootstrap required, opened a new log
I20260925 15:21:51.294088 22474 ts_tablet_manager.cc:1397] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260925 15:21:51.295001 22475 tablet_bootstrap.cc:492] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: No bootstrap required, opened a new log
I20260925 15:21:51.295259 22475 ts_tablet_manager.cc:1397] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260925 15:21:51.305763 22478 tablet_bootstrap.cc:492] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Bootstrap starting.
I20260925 15:21:51.307659 22474 raft_consensus.cc:348] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.308195 22474 raft_consensus.cc:374] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:51.308327 22474 raft_consensus.cc:729] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffb800fc4f6489cbe9130a39125d7d1, State: Initialized, Role: FOLLOWER
I20260925 15:21:51.308732 22475 raft_consensus.cc:348] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.309181 22475 raft_consensus.cc:374] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:51.309278 22475 raft_consensus.cc:729] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a379b95789214efe9d513b320872b0c3, State: Initialized, Role: FOLLOWER
I20260925 15:21:51.309908 22475 consensus_queue.cc:260] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.310673 22478 tablet_bootstrap.cc:654] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:51.310566 22474 consensus_queue.cc:260] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.312252 22478 log.cc:824] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:51.335134 22174 heartbeater.cc:503] Master 127.8.226.126:37495 was elected leader, sending a full tablet report...
I20260925 15:21:51.335539 22452 heartbeater.cc:503] Master 127.8.226.126:37495 was elected leader, sending a full tablet report...
I20260925 15:21:51.335542 22475 ts_tablet_manager.cc:1428] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Time spent starting tablet: real 0.040s user 0.023s sys 0.015s
I20260925 15:21:51.335959 22474 ts_tablet_manager.cc:1428] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Time spent starting tablet: real 0.042s user 0.023s sys 0.013s
I20260925 15:21:51.339053 22478 tablet_bootstrap.cc:492] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: No bootstrap required, opened a new log
I20260925 15:21:51.339430 22478 ts_tablet_manager.cc:1397] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.022s
I20260925 15:21:51.350358 22478 raft_consensus.cc:348] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.350885 22478 raft_consensus.cc:374] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:51.350998 22478 raft_consensus.cc:729] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5262a57831434f78bad321ef8f5923c8, State: Initialized, Role: FOLLOWER
I20260925 15:21:51.351650 22478 consensus_queue.cc:260] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.352841 22318 heartbeater.cc:503] Master 127.8.226.126:37495 was elected leader, sending a full tablet report...
I20260925 15:21:51.353585 22478 ts_tablet_manager.cc:1428] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260925 15:21:51.377904 22319 tablet.cc:2369] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:51.431460 22479 raft_consensus.cc:482] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:21:51.431787 22479 raft_consensus.cc:504] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.433867 22479 leader_election.cc:288] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5262a57831434f78bad321ef8f5923c8 (127.8.226.66:34209), 0ffb800fc4f6489cbe9130a39125d7d1 (127.8.226.65:39801)
I20260925 15:21:51.442027 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.434079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:34209 (local address 127.8.226.67:55881)
0925 15:21:51.434744 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.434763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:51.434902 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:51.435638 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:51.435644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:51.435669 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:51.435854 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.435865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.437337 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.437343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.439475 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.439490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.441249 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.441254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.441485 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.441520 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:21:51.441631 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:21:51.441761 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551}
I20260925 15:21:51.442692 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.434388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:55881 (local address 127.8.226.66:34209)
0925 15:21:51.434585 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.434593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.435145 (+ 552us) server_negotiation.cc:407] Connection header received
0925 15:21:51.435283 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.435288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.435369 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.435461 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:51.436026 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.437168 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.439664 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.440921 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.441094 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.441166 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.442291 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.442424 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:21:51.443403 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.435469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:34441 (local address 127.8.226.65:39801)
0925 15:21:51.435661 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.435667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.435685 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:21:51.435764 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.435769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.435836 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.435929 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:51.436830 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.437895 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.440417 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.441799 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.441926 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.441996 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.443038 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.443171 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:21:51.443606 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795409c15243459c9ceeb0aab5a8e8a1" candidate_uuid: "a379b95789214efe9d513b320872b0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5262a57831434f78bad321ef8f5923c8" is_pre_election: true
I20260925 15:21:51.444141 22273 raft_consensus.cc:2454] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a379b95789214efe9d513b320872b0c3 in term 0.
I20260925 15:21:51.444437 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.434468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.8.226.67:34441)
0925 15:21:51.435002 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.435020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:51.435145 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:51.436240 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:51.436245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:51.436270 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:51.436627 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.436639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.438096 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.438101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.440234 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:51.440248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.442405 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.442412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:51.442584 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.442616 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:21:51.443760 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0925 15:21:51.443960 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":6,"thread_start_us":1585,"threads_started":1}
I20260925 15:21:51.444893 22343 leader_election.cc:302] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5262a57831434f78bad321ef8f5923c8, a379b95789214efe9d513b320872b0c3; no voters:
I20260925 15:21:51.445421 22479 raft_consensus.cc:2781] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:21:51.445564 22479 raft_consensus.cc:482] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:21:51.445672 22479 raft_consensus.cc:3037] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:51.446501 22129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795409c15243459c9ceeb0aab5a8e8a1" candidate_uuid: "a379b95789214efe9d513b320872b0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" is_pre_election: true
I20260925 15:21:51.446990 22129 raft_consensus.cc:2454] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a379b95789214efe9d513b320872b0c3 in term 0.
I20260925 15:21:51.448853 22479 raft_consensus.cc:504] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.449720 22479 leader_election.cc:288] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [CANDIDATE]: Term 1 election: Requested vote from peers 5262a57831434f78bad321ef8f5923c8 (127.8.226.66:34209), 0ffb800fc4f6489cbe9130a39125d7d1 (127.8.226.65:39801)
I20260925 15:21:51.450510 22129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795409c15243459c9ceeb0aab5a8e8a1" candidate_uuid: "a379b95789214efe9d513b320872b0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffb800fc4f6489cbe9130a39125d7d1"
I20260925 15:21:51.450718 22129 raft_consensus.cc:3037] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:51.453888 22129 raft_consensus.cc:2454] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a379b95789214efe9d513b320872b0c3 in term 1.
I20260925 15:21:51.454284 22343 leader_election.cc:302] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffb800fc4f6489cbe9130a39125d7d1, a379b95789214efe9d513b320872b0c3; no voters:
I20260925 15:21:51.454684 22273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795409c15243459c9ceeb0aab5a8e8a1" candidate_uuid: "a379b95789214efe9d513b320872b0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5262a57831434f78bad321ef8f5923c8"
I20260925 15:21:51.454888 22273 raft_consensus.cc:3037] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:51.458035 22273 raft_consensus.cc:2454] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a379b95789214efe9d513b320872b0c3 in term 1.
I20260925 15:21:51.458426 22479 raft_consensus.cc:2781] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:51.458899 22479 raft_consensus.cc:686] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 1 LEADER]: Becoming Leader. State: Replica: a379b95789214efe9d513b320872b0c3, State: Running, Role: LEADER
I20260925 15:21:51.459764 22479 consensus_queue.cc:237] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } }
I20260925 15:21:51.465273 21977 catalog_manager.cc:5495] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a379b95789214efe9d513b320872b0c3 (127.8.226.67). New cstate: current_term: 1 leader_uuid: "a379b95789214efe9d513b320872b0c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a379b95789214efe9d513b320872b0c3" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 } health_report { overall_health: UNKNOWN } } }
W20260925 15:21:51.488426 22175 tablet.cc:2369] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:51.519186 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.514395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.8.226.67:43273)
0925 15:21:51.514569 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.514573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.514589 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:21:51.514714 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.514717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.514805 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.514923 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:21:51.515261 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.515953 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.517299 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.517683 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.517802 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.518255 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:21:51.518744 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:21:51.518856 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.518931 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.518990 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:21:51.527058 22129 raft_consensus.cc:1261] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 1 FOLLOWER]: Refusing update from remote peer a379b95789214efe9d513b320872b0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:51.527469 22273 raft_consensus.cc:1261] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 1 FOLLOWER]: Refusing update from remote peer a379b95789214efe9d513b320872b0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:51.528139 22485 consensus_queue.cc:1035] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ffb800fc4f6489cbe9130a39125d7d1" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:51.528594 22479 consensus_queue.cc:1035] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5262a57831434f78bad321ef8f5923c8" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 34209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:51.541635 22494 mvcc.cc:205] Tried to move back new op lower bound from 7333272418405572608 to 7333272418144313344. Current Snapshot: MvccSnapshot[applied={T|T < 7333272418405572608}]
I20260925 15:21:51.542997 22495 mvcc.cc:205] Tried to move back new op lower bound from 7333272418405572608 to 7333272418144313344. Current Snapshot: MvccSnapshot[applied={T|T < 7333272418405572608}]
I20260925 15:21:51.554160 21976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:51.554445 21976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:51.558262 21976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9e6d8006d824bb7bd893a48eb9769ff: Sending DeleteTablet for 3 replicas of tablet 795409c15243459c9ceeb0aab5a8e8a1
I20260925 15:21:51.559427 22387 tablet_service.cc:1514] Processing DeleteTablet for tablet 795409c15243459c9ceeb0aab5a8e8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:51 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:57756
I20260925 15:21:51.559584 22109 tablet_service.cc:1514] Processing DeleteTablet for tablet 795409c15243459c9ceeb0aab5a8e8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:51 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:42366
I20260925 15:21:51.559615 22253 tablet_service.cc:1514] Processing DeleteTablet for tablet 795409c15243459c9ceeb0aab5a8e8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:51 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:42316
I20260925 15:21:51.571715 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22014
I20260925 15:21:51.571835 22505 tablet_replica.cc:330] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: stopping tablet replica
I20260925 15:21:51.572389 22505 raft_consensus.cc:2229] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:51.573010 22505 raft_consensus.cc:2258] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:51.573972 22504 tablet_replica.cc:330] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: stopping tablet replica
I20260925 15:21:51.574458 22504 raft_consensus.cc:2229] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:51.574987 22504 raft_consensus.cc:2258] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:51.575217 22503 tablet_replica.cc:330] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: stopping tablet replica
I20260925 15:21:51.576167 22503 raft_consensus.cc:2229] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:21:51.576786 22503 raft_consensus.cc:2258] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:51.577122 22505 ts_tablet_manager.cc:1905] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:51.578644 22503 ts_tablet_manager.cc:1905] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:51.583900 22505 ts_tablet_manager.cc:1918] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:51.584048 22505 log.cc:1196] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/795409c15243459c9ceeb0aab5a8e8a1
I20260925 15:21:51.584580 22505 ts_tablet_manager.cc:1939] T 795409c15243459c9ceeb0aab5a8e8a1 P 5262a57831434f78bad321ef8f5923c8: Deleting consensus metadata
I20260925 15:21:51.586216 22504 ts_tablet_manager.cc:1905] T 795409c15243459c9ceeb0aab5a8e8a1 P 0ffb800fc4f6489cbe9130a39125d7d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:51.586619 22503 ts_tablet_manager.cc:1918] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:51.586804 22503 log.cc:1196] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/795409c15243459c9ceeb0aab5a8e8a1
I20260925 15:21:51.587352 22503 ts_tablet_manager.cc:1939] T 795409c15243459c9ceeb0aab5a8e8a1 P a379b95789214efe9d513b320872b0c3: Deleting consensus metadata
I20260925 15:21:51.589083 21963 catalog_manager.cc:4841] TS a379b95789214efe9d513b320872b0c3 (127.8.226.67:43273): tablet 795409c15243459c9ceeb0aab5a8e8a1 (table test-table [id=22650d9f0df84a83b3e885ef94dcb779]) successfully deleted
I20260925 15:21:51.590020 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.573292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.8.226.65:39801)
0925 15:21:51.573439 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.573444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.573461 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:21:51.573538 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.573545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.573646 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.573770 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:51.575402 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.576280 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.578201 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.578611 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.578763 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.582260 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:51.582270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:51.582282 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:51.582317 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:51.585736 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.586578 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.586586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.586608 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.586798 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.587286 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.587292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.587295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.587767 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:51.588090 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.589320 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.589582 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:21:51.590716 22169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:51.592039 21964 catalog_manager.cc:4841] TS 5262a57831434f78bad321ef8f5923c8 (127.8.226.66:34209): tablet 795409c15243459c9ceeb0aab5a8e8a1 (table test-table [id=22650d9f0df84a83b3e885ef94dcb779]) successfully deleted
I20260925 15:21:51.706244 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22014
I20260925 15:21:51.722851 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22178
W20260925 15:21:51.722846 21964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:39801 (error 108)
I20260925 15:21:51.724802 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.724303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42390)
0925 15:21:51.724490 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.724625 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:21:51.725132 21964 catalog_manager.cc:4584] TS 0ffb800fc4f6489cbe9130a39125d7d1 (127.8.226.65:39801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 795409c15243459c9ceeb0aab5a8e8a1: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
I20260925 15:21:51.739259 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.724060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.8.226.66:34209)
0925 15:21:51.724269 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.724275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.724297 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:21:51.724366 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.724373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.724471 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.724587 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:51.726212 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.727442 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.728633 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.729561 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.729717 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.732248 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:51.732265 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:51.732286 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:51.732330 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:51.736417 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.737209 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.737217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.737219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.737359 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.737740 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.737748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.737754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.738092 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:51.738391 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.738666 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.738909 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:21:51.739867 22313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:51.778712 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.778263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42394)
0925 15:21:51.778482 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.778565 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:21:51.779731 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.779402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42410)
0925 15:21:51.779546 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.779620 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:21:51.840421 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.839849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42414)
0925 15:21:51.840104 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.840211 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 15:21:51.841725 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.841267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42422)
0925 15:21:51.841445 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.841542 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:21:51.848927 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22178
I20260925 15:21:51.865579 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22322
I20260925 15:21:51.882268 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.866338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.8.226.67:43273)
0925 15:21:51.866558 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:51.866565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:51.866588 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:21:51.866751 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:51.866757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:51.866858 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:51.866979 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:51.868378 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.869859 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.871264 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:51.872000 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:51.872170 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:51.874751 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:51.874765 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:51.874786 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:51.874840 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:51.879220 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.879934 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.879944 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.879949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.880143 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:51.880509 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:51.880514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:51.880518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:51.881012 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:51.881384 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:51.881548 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:21:51.881807 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 15:21:51.883155 22447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:51.922829 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.922282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42426)
0925 15:21:51.922525 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.922623 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 15:21:51.923975 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:51.923609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42440)
0925 15:21:51.923811 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:51.923877 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 15:21:52.002856 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22322
I20260925 15:21:52.021106 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 21944
I20260925 15:21:52.022361 22007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:52.143208 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 21944
I20260925 15:21:52.143244 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:52.142382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39801 (local address 127.0.0.1:42442)
0925 15:21:52.142913 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:52.143072 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
I20260925 15:21:52.166749 9097 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:21:52.502318 9097 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:21:52.502547 9097 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:21:52.847657 9097 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:21:52.847 UTC [20996] LOG: received smart shutdown request
2026-09-25 15:21:52.850 UTC [20996] LOG: background worker "logical replication launcher" (PID 21005) exited with exit code 1
2026-09-25 15:21:52.851 UTC [21000] LOG: shutting down
2026-09-25 15:21:52.990 UTC [20996] LOG: database system is shut down
2026-09-25T15:21:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49915 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:53 dist-test-slave-8vw7 krb5kdc[22519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:53 dist-test-slave-8vw7 krb5kdc[22519](info): set up 2 sockets
Sep 25 15:21:53 dist-test-slave-8vw7 krb5kdc[22519](info): commencing operation
krb5kdc: starting...
W20260925 15:21:54.197681 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:21:54 dist-test-slave-8vw7 krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:21:54.410207 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:36283
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:36283,127.8.226.125:46097,127.8.226.124:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:54.721020 22532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:54.721282 22532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:54.721457 22532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:54.728602 22532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:54.728699 22532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:54.728796 22532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:54.728895 22532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:54.728992 22532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:54.738343 22532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:36283,127.8.226.125:46097,127.8.226.124:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:54.739537 22532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:54.740754 22532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:54.752065 22538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:54.752038 22537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:54.752658 22540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:54.752939 22532 server_base.cc:1034] running on GCE node
I20260925 15:21:54.753711 22532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:54.754956 22532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:54.756174 22532 hybrid_clock.cc:648] HybridClock initialized: now 1790349714756143 us; error 42 us; skew 500 ppm
I20260925 15:21:54.756517 22532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:54 dist-test-slave-8vw7 krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:54.759490 22532 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:21:54.760871 22532 webserver.cc:460] Webserver started at http://127.8.226.126:40253/ using document root <none> and password file <none>
I20260925 15:21:54.761545 22532 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:54.761706 22532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:54.761987 22532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:54.764732 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "3fce8a7b63cf462ea4395e5e96657ac4"
format_stamp: "Formatted at 2026-09-25 15:21:54 on dist-test-slave-8vw7"
server_key: "f2f2206c3445b4e55264d49282cbc371"
server_key_iv: "6b3fe588e572d4011eaea426c7b6c880"
server_key_version: "encryptionkey@0"
I20260925 15:21:54.765552 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "3fce8a7b63cf462ea4395e5e96657ac4"
format_stamp: "Formatted at 2026-09-25 15:21:54 on dist-test-slave-8vw7"
server_key: "f2f2206c3445b4e55264d49282cbc371"
server_key_iv: "6b3fe588e572d4011eaea426c7b6c880"
server_key_version: "encryptionkey@0"
I20260925 15:21:54.770411 22532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:21:54.773675 22547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:54.774825 22532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:21:54.774998 22532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "3fce8a7b63cf462ea4395e5e96657ac4"
format_stamp: "Formatted at 2026-09-25 15:21:54 on dist-test-slave-8vw7"
server_key: "f2f2206c3445b4e55264d49282cbc371"
server_key_iv: "6b3fe588e572d4011eaea426c7b6c880"
server_key_version: "encryptionkey@0"
I20260925 15:21:54.775187 22532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:54.797586 22532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:54.801407 22532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:54.801681 22532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:54.819880 22532 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:36283
I20260925 15:21:54.819916 22598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:36283 every 8 connection(s)
I20260925 15:21:54.821213 22532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:54.824743 22599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:54.830200 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22532
I20260925 15:21:54.830430 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:21:54.831251 9097 external_mini_cluster.cc:1428] Setting key d8d80a461e6f9ecf784efeb8a8e1e95b
I20260925 15:21:54.831696 22599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:21:54 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.125@KRBTEST.COM" created.
I20260925 15:21:54.856351 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.835678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36283 (local address 127.0.0.1:51116)
0925 15:21:54.836637 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.836705 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:54.838774 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:54.839580 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:54.839586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:54.840362 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:54.840841 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:54.840866 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:54.842475 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:54.842484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:54.844480 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:54.844498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:54.845481 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:54.845491 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:54.845730 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:54.846442 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:54.846475 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:54.849096 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:54.852203 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:54.852215 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:54.852237 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:54.852604 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:54.853050 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:54.853057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:54.853060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:54.853211 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:54.853988 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:54.853995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:54.854199 (+ 204us) client_negotiation.cc:769] Sending connection context
0925 15:21:54.854435 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 15:21:54.855048 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":198,"thread_start_us":415,"threads_started":1}
I20260925 15:21:54.856325 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.837106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.8.226.126:36283)
0925 15:21:54.837761 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:54.837775 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:54.837830 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:21:54.839134 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:54.839140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:54.839361 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:54.839560 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:54.841093 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:54.842290 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:54.844627 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:54.845250 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:54.845380 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:54.849326 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:54.849334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:54.849361 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:54.849391 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:54.852076 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:54.852750 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:54.852757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:54.852759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:54.852893 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:54.853330 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:54.853335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:54.853338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:54.853855 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:54.854058 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:54.854551 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 15:21:54.854942 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":427,"thread_start_us":267,"threads_started":1}
I20260925 15:21:54.859568 22599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:21:54.860694 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.860205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50826)
0925 15:21:54.860408 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.860535 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 15:21:54.861012 22549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
I20260925 15:21:54.862576 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.862135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50828)
0925 15:21:54.862303 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.862387 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:21:54.862816 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:46097: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab.
Entry for principal kudu/127.8.226.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab.
I20260925 15:21:54.882386 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:46097
--webserver_interface=127.8.226.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab
--principal=kudu/127.8.226.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:36283,127.8.226.125:46097,127.8.226.124:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:21:54.912264 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } attempt: 1
I20260925 15:21:54.913461 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.913009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50842)
0925 15:21:54.913206 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.913325 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:21:54.914532 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.914170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50844)
0925 15:21:54.914314 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.914397 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:21:54.914716 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:46097: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
I20260925 15:21:54.982937 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } attempt: 2
I20260925 15:21:54.984189 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.983678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50850)
0925 15:21:54.983944 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.984042 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:21:54.985270 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:54.984861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50852)
0925 15:21:54.985061 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:54.985139 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:21:54.985461 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:46097: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
I20260925 15:21:55.076690 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } attempt: 3
I20260925 15:21:55.077824 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.077386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50858)
0925 15:21:55.077583 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.077697 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:21:55.078871 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.078489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50872)
0925 15:21:55.078626 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.078702 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:21:55.079036 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:46097: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
W20260925 15:21:55.207847 22606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:55.208186 22606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:55.208343 22606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:55.215595 22606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:55.215736 22606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:55.215886 22606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:55.215976 22606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:55.216058 22606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:55.222272 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } attempt: 4
I20260925 15:21:55.223392 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.222879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50876)
0925 15:21:55.223175 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.223265 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260925 15:21:55.224377 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.224023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50888)
0925 15:21:55.224166 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.224262 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:21:55.224562 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:46097: Network error: Client connection negotiation failed: client connection to 127.8.226.125:46097: connect: Connection refused (error 111)
I20260925 15:21:55.226481 22606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:36283,127.8.226.125:46097,127.8.226.124:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab
--principal=kudu/127.8.226.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.226.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:55.227824 22606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:55.229115 22606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:55.240696 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:55.240760 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:55.240773 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:55.241091 22606 server_base.cc:1034] running on GCE node
I20260925 15:21:55.242098 22606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:55.243480 22606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:55.244710 22606 hybrid_clock.cc:648] HybridClock initialized: now 1790349715244650 us; error 46 us; skew 500 ppm
I20260925 15:21:55.245090 22606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:55.258703 22606 init.cc:377] Logged in from keytab as kudu/127.8.226.125@KRBTEST.COM (short username kudu)
I20260925 15:21:55.260315 22606 webserver.cc:460] Webserver started at http://127.8.226.125:35493/ using document root <none> and password file <none>
I20260925 15:21:55.261202 22606 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:55.261361 22606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:55.261677 22606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:55.264796 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data/instance:
uuid: "163fa78a168f44489eb40df8fad5f040"
format_stamp: "Formatted at 2026-09-25 15:21:55 on dist-test-slave-8vw7"
server_key: "65374cf37cea5193a5c41ffa3b29bf28"
server_key_iv: "b127aeac45a8354278520a9ac8f4ed24"
server_key_version: "encryptionkey@0"
I20260925 15:21:55.265693 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal/instance:
uuid: "163fa78a168f44489eb40df8fad5f040"
format_stamp: "Formatted at 2026-09-25 15:21:55 on dist-test-slave-8vw7"
server_key: "65374cf37cea5193a5c41ffa3b29bf28"
server_key_iv: "b127aeac45a8354278520a9ac8f4ed24"
server_key_version: "encryptionkey@0"
I20260925 15:21:55.270625 22606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:21:55.274047 22621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:55.275363 22606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:21:55.275547 22606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal
uuid: "163fa78a168f44489eb40df8fad5f040"
format_stamp: "Formatted at 2026-09-25 15:21:55 on dist-test-slave-8vw7"
server_key: "65374cf37cea5193a5c41ffa3b29bf28"
server_key_iv: "b127aeac45a8354278520a9ac8f4ed24"
server_key_version: "encryptionkey@0"
I20260925 15:21:55.275755 22606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:55.297194 22606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:55.301527 22606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:55.301819 22606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:55.321164 22606 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.125:46097
I20260925 15:21:55.321184 22672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.125:46097 every 8 connection(s)
I20260925 15:21:55.322525 22606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
I20260925 15:21:55.323266 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22606
I20260925 15:21:55.323460 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-1/wal/instance
I20260925 15:21:55.323952 9097 external_mini_cluster.cc:1428] Setting key 4f1d66d956c07bb98fee35d011039502
I20260925 15:21:55.326964 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:55.333544 22673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.226.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.124@KRBTEST.COM" created.
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:55.354900 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.336277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.8.226.126:36283)
0925 15:21:55.336498 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.336504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.337746 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 15:21:55.339715 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.339722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.339822 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.339952 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.341436 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.342563 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.344734 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.345215 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.345385 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.349343 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.349348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.349351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.349400 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.351978 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.352685 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.352691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.352694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.352812 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.353252 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.353259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.353266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.353525 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.353677 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.354331 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.354569 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:21:55.355739 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.336724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36283 (local address 127.0.0.1:51126)
0925 15:21:55.337560 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.337617 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.339434 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.340144 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.340152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.340650 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.341267 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.341284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.342710 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.342713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.344606 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.344618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.345451 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.345461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.345736 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.346482 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.346523 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.349139 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.352120 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.352129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.352144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.352561 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.352921 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.352927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.352930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.353127 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.353661 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.353669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.353977 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.354228 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.354677 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":407,"threads_started":1}
I20260925 15:21:55.357400 22673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab.
Entry for principal kudu/127.8.226.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab.
I20260925 15:21:55.371812 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:43687
--webserver_interface=127.8.226.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab
--principal=kudu/127.8.226.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:36283,127.8.226.125:46097,127.8.226.124:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:21:55.372144 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.357937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:50898)
0925 15:21:55.358080 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.358099 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.358237 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.359242 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.359247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.359652 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.359763 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.359770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.361082 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.361088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.361881 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.361891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.362658 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.362664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.362870 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.363405 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.363420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.366601 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.369795 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.369800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.369804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.370146 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.370541 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.370545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.370549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.370705 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.371249 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.371256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.371394 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.371525 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.371768 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:21:55.372225 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.358261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.8.226.125:46097)
0925 15:21:55.358859 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.358864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.358886 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:21:55.358921 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.358926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.359080 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.359242 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.359908 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.360916 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.362028 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.362524 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.362653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.366824 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.366839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.366854 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.366890 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.369658 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.370302 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.370311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.370317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.370429 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.370804 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.370812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.370815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.371129 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.371336 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.371666 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.371863 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":329,"threads_started":1}
I20260925 15:21:55.374456 22673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:21:55.375511 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.375094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:42444)
0925 15:21:55.375250 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.375363 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:21:55.375845 22622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.377424 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.377128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:42456)
0925 15:21:55.377254 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.377327 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:21:55.377653 22673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.427066 22673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 1
I20260925 15:21:55.428218 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.427781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32908)
0925 15:21:55.427963 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.428076 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:21:55.429216 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.428889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32918)
0925 15:21:55.429065 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.429124 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:21:55.429406 22673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.497606 22673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 2
I20260925 15:21:55.498806 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.498363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32924)
0925 15:21:55.498558 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.498648 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:21:55.499835 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.499499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32938)
0925 15:21:55.499653 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.499720 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:21:55.500015 22673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.523830 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } attempt: 5
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:21:55.536814 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.524564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:40092)
0925 15:21:55.524763 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.524784 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.524915 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.525320 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.525324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.525714 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.525899 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.525908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.526878 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.526885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.527747 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.527759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.528461 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.528466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.528653 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.529314 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.529332 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.531571 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.534528 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.534535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.534538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.534886 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.535240 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.535248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.535252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.535388 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.535880 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.535885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.536036 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.536169 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.536464 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:21:55.536814 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.524671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.8.226.125:46097)
0925 15:21:55.524884 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.524890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.524910 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:21:55.525099 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.525103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.525180 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.525286 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.526049 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.526746 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.527886 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.528310 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.528435 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.531767 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.531772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.531775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.531806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.534388 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.534997 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.535005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.535008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.535128 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.535490 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.535497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.535500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.535765 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.535916 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.536285 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.536498 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:21:55.538286 22599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:21:55.539405 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.539028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32950)
0925 15:21:55.539185 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.539266 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:21:55.540611 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.540093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32966)
0925 15:21:55.540293 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.540427 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 15:21:55.540891 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.591199 22673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 3
I20260925 15:21:55.592173 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 1
I20260925 15:21:55.592366 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.591902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32972)
0925 15:21:55.592103 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.592189 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:21:55.593372 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.592816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32988)
0925 15:21:55.593051 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.593178 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:21:55.593499 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.593069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:32996)
0925 15:21:55.593287 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.593360 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260925 15:21:55.593755 22673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.594427 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.594029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33010)
0925 15:21:55.594195 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.594285 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:21:55.594643 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.662873 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 2
I20260925 15:21:55.664126 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.663509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33016)
0925 15:21:55.663785 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.663912 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:21:55.665208 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.664775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33028)
0925 15:21:55.664929 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.665043 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:21:55.665431 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
W20260925 15:21:55.694772 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:55.695014 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:55.695093 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:55.701719 22680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:55.701812 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:55.701897 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:55.701980 22680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:55.702036 22680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:55.710819 22680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:36283,127.8.226.125:46097,127.8.226.124:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab
--principal=kudu/127.8.226.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.226.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:55.711866 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:55.713094 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:55.724062 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:55.724108 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:55.724143 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:55.724757 22680 server_base.cc:1034] running on GCE node
I20260925 15:21:55.725534 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:55.726828 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:55.728044 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790349715728010 us; error 47 us; skew 500 ppm
I20260925 15:21:55.728381 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:55.731374 22680 init.cc:377] Logged in from keytab as kudu/127.8.226.124@KRBTEST.COM (short username kudu)
I20260925 15:21:55.732698 22680 webserver.cc:460] Webserver started at http://127.8.226.124:44545/ using document root <none> and password file <none>
I20260925 15:21:55.733431 22680 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:55.733569 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:55.733852 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:55.736598 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data/instance:
uuid: "04bb4ea92c804ec6ae80d964d515b557"
format_stamp: "Formatted at 2026-09-25 15:21:55 on dist-test-slave-8vw7"
server_key: "6ccece018b7b105074859206a9ec27e9"
server_key_iv: "08deeeb973d21ad3329e390586227536"
server_key_version: "encryptionkey@0"
I20260925 15:21:55.737003 22673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 4
I20260925 15:21:55.737540 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal/instance:
uuid: "04bb4ea92c804ec6ae80d964d515b557"
format_stamp: "Formatted at 2026-09-25 15:21:55 on dist-test-slave-8vw7"
server_key: "6ccece018b7b105074859206a9ec27e9"
server_key_iv: "08deeeb973d21ad3329e390586227536"
server_key_version: "encryptionkey@0"
I20260925 15:21:55.738059 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.737608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33040)
0925 15:21:55.737836 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.737915 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:21:55.738932 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.738668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33046)
0925 15:21:55.738786 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.738845 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:21:55.739159 22673 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.742698 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:21:55.745815 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:55.747012 22680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:21:55.747172 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal
uuid: "04bb4ea92c804ec6ae80d964d515b557"
format_stamp: "Formatted at 2026-09-25 15:21:55 on dist-test-slave-8vw7"
server_key: "6ccece018b7b105074859206a9ec27e9"
server_key_iv: "08deeeb973d21ad3329e390586227536"
server_key_version: "encryptionkey@0"
I20260925 15:21:55.747380 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:55.768085 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:55.771631 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 3
I20260925 15:21:55.772192 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:55.772445 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:55.772643 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.772190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33058)
0925 15:21:55.772388 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.772464 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:21:55.773836 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.773390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33064)
0925 15:21:55.773636 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.773709 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260925 15:21:55.774101 22599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:43687: Network error: Client connection negotiation failed: client connection to 127.8.226.124:43687: connect: Connection refused (error 111)
I20260925 15:21:55.790125 22680 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.124:43687
I20260925 15:21:55.790167 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.124:43687 every 8 connection(s)
I20260925 15:21:55.791344 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
I20260925 15:21:55.793300 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22680
I20260925 15:21:55.793488 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349523192742-9097-0/minicluster-data/master-2/wal/instance
I20260925 15:21:55.793967 9097 external_mini_cluster.cc:1428] Setting key 46e4e42ba1513a7a5eafb82c83c60dc3
I20260925 15:21:55.795817 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:55.805389 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:55.817432 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.795890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.8.226.124:43687)
0925 15:21:55.796707 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.796721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.796768 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:21:55.798993 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.799019 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.799543 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.799987 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.801356 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.803461 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.804850 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.805869 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.806052 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.810577 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.810584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.810594 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.810628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.813337 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.814043 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.814049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.814054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.814216 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.814603 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.814608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.814612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.815086 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.815397 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.815823 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.816226 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":339,"threads_started":1}
I20260925 15:21:55.822342 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.808318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:36283 (local address 127.0.0.1:42550)
0925 15:21:55.809084 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.809126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.809261 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.809667 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.809671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.810179 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.810414 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.810421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.811525 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.811536 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.812761 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.812772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.813571 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.813577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.813749 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.814291 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.814307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.816334 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.819683 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.819690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.819695 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.820191 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.820684 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.820689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.820692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.820831 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.821364 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.821371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.821542 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.821707 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.821976 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":411,"threads_started":1}
I20260925 15:21:55.822351 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.808386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.8.226.126:36283)
0925 15:21:55.808578 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.808585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.809227 (+ 642us) server_negotiation.cc:407] Connection header received
0925 15:21:55.809384 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.809390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.809467 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.809576 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.810559 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.811376 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.812896 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.813364 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.813534 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.816503 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.816509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.816513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.816549 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.819533 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.820343 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.820351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.820354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.820521 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.820944 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.820971 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.820976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.821221 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.821395 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.821820 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.822006 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:21:55.823776 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:21:55.839641 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.820291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.8.226.126:36283)
0925 15:21:55.822542 (+ 2251us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.822549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.822566 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:21:55.822611 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.822614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.822683 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.822799 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.824381 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.825210 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.826942 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.827539 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.827643 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.832930 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.832936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.832941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.833010 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.837088 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.837698 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.837706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.837723 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.837857 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.838237 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.838240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.838242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.838577 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.838796 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.839148 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.839351 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2160,"thread_start_us":16144,"threads_started":1}
I20260925 15:21:55.840296 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.824352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:46097 (local address 127.0.0.1:40108)
0925 15:21:55.825433 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.825449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.825597 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.826068 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.826073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.826483 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.826660 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.826671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.827973 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.827979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.829344 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.829360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.830630 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.830647 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.830884 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.831510 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.831535 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.834092 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.837791 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.837794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.837796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.838249 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.838661 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.838664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.838668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.838850 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.839571 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.839574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.839707 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.839863 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.840094 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978}
I20260925 15:21:55.840727 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.824441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.8.226.125:46097)
0925 15:21:55.824654 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.824662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.825512 (+ 850us) server_negotiation.cc:407] Connection header received
0925 15:21:55.825740 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.825746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.825817 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.825883 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.826835 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.827811 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.829514 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.830264 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.830373 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.834326 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.834332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.834336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.834378 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.837659 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.838371 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.838376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.838379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.838514 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.838970 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.838973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.838975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.839294 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.839433 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.839983 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.840239 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:21:55.841621 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:21:55.855060 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.842352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33086)
0925 15:21:55.842532 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.842551 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.842658 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.842954 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.842958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.843371 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.843490 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.843498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.844717 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.844731 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.845975 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.845985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.846820 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.846829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.846991 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.847397 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.847420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.849707 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.852714 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.852719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.852723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.853216 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.853547 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.853554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.853560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.853722 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.854167 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.854171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.854296 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.854416 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.854693 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:21:55.855186 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.842390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.8.226.124:43687)
0925 15:21:55.842544 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.842551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.842599 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:21:55.842756 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.842759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.842805 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.842874 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.843641 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.844600 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.846087 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.846656 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.846737 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.849869 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.849873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.849876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.849901 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.852603 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.853312 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.853315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.853317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.853438 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.853829 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.853832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.853834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.854050 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.854200 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.854501 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.854836 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":48}
I20260925 15:21:55.856297 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.841327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.8.226.125:46097)
0925 15:21:55.841486 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.841491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.841596 (+ 105us) server_negotiation.cc:407] Connection header received
0925 15:21:55.841704 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.841708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.841776 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.841868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.843248 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.844265 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.845842 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.846322 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.846437 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.851091 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.851097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.851101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.851139 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.854004 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.854605 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.854610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.854613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.854740 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.855100 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.855105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.855108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.855402 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.855665 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.855888 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.856028 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:21:55.860491 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: Bootstrap starting.
I20260925 15:21:55.860564 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:21:55.863770 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:55.864775 22747 log.cc:824] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:55.875396 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: No bootstrap required, opened a new log
I20260925 15:21:55.884498 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.865663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.8.226.126:36283)
0925 15:21:55.865806 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.865811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.865830 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:21:55.865895 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.865902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.865989 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.866108 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.869118 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.870281 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.872274 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.872921 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.873056 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.876638 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.876647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.876654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.876697 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.880878 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.881713 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.881718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.881722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.881888 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.882351 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.882357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.882363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.882703 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.882967 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.883886 (+ 919us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.884106 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:21:55.885711 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.866280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.8.226.125:46097)
0925 15:21:55.866588 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.866596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.868345 (+ 1749us) server_negotiation.cc:407] Connection header received
0925 15:21:55.868529 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.868534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.868601 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.868688 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.870773 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.871791 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.873879 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.874554 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.874652 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.878340 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.878349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.878356 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.878392 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.881790 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.882599 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.882604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.882608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.882759 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.883216 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.883221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.883224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.883602 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.883836 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.884376 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.884599 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260925 15:21:55.886251 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.866119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.8.226.124:43687)
0925 15:21:55.866461 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.866467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.867190 (+ 723us) server_negotiation.cc:407] Connection header received
0925 15:21:55.867400 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.867408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.867495 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.867595 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.870321 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.871701 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.874095 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.874760 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.874851 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.878869 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.878875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.878880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.878915 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.882521 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.883402 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.883408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.883412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.883580 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.884036 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.884041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.884045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.884363 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.884838 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.885434 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.885675 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260925 15:21:55.887284 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:55.887692 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:55.887831 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bb4ea92c804ec6ae80d964d515b557, State: Initialized, Role: FOLLOWER
I20260925 15:21:55.888417 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:55.889259 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:55.889611 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:55.890259 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:21:55.899333 22772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:21:55.899454 22772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:21:55.897464 22680 master_runner.cc:186] Error getting master registration for 127.8.226.126:36283: OK, instance_id { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" instance_seqno: 1790349714814740 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:21:55.908406 22680 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:21:55.951334 22599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 4
Sep 25 15:21:55 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
I20260925 15:21:55.966020 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.951903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.8.226.124:43687)
0925 15:21:55.952084 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:55.952088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:55.952187 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:21:55.952328 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:55.952332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:55.952406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:55.952486 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:55.953450 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.954401 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.956006 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.956409 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.956528 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.959690 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:55.959697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:55.959702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:55.959740 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:55.963121 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.963736 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.963742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.963746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.963906 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:55.964268 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:55.964273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:55.964277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:55.964659 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:55.964815 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:55.965422 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 15:21:55.965624 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:21:55.966789 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:55.951892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33090)
0925 15:21:55.952097 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:55.952118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:55.952232 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:55.952544 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:55.952549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:55.953088 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:55.953284 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.953296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.954589 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.954594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.955854 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:55.955867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:55.956546 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:55.956552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:55.956733 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:55.957356 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:55.957372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:55.959504 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:55.963238 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.963241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.963244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.963630 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.964022 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:55.964025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:55.964027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:55.964143 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:55.964782 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:55.964785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:55.965007 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 15:21:55.965164 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 15:21:55.966419 (+ 1255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:21:55.972312 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: Bootstrap starting.
I20260925 15:21:55.976151 22599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:55.977383 22599 log.cc:824] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:55.981245 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: No bootstrap required, opened a new log
I20260925 15:21:55.987903 22599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:55.988164 22599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:55.988322 22599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fce8a7b63cf462ea4395e5e96657ac4, State: Initialized, Role: FOLLOWER
I20260925 15:21:55.988859 22599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:55.989780 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:55.990141 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:55.990938 22599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:21:56.001286 22786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:21:56.001399 22786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:21:56.000582 22532 master_runner.cc:186] Error getting master registration for 127.8.226.125:46097: OK, instance_id { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" instance_seqno: 1790349715315879 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:21:56.012477 22532 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:21:56.024646 22701 tablet.cc:2369] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:56.038406 22673 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } attempt: 5
Sep 25 15:21:56 dist-test-slave-8vw7 krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
W20260925 15:21:56.054239 22553 tablet.cc:2369] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:56.055389 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:56.038992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:43687 (local address 127.0.0.1:33094)
0925 15:21:56.039241 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:56.039260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:56.039372 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:56.039757 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:56.039762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:56.040261 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:56.040463 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:56.040476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:56.041933 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:56.041938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:56.043346 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:56.043377 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:56.044673 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:56.044679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:56.044934 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:56.045676 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:56.045715 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:56.048401 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:56.052525 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:56.052530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:56.052534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:56.053035 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:56.053449 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:56.053454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:56.053458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:56.053612 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:56.054334 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:56.054339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:56.054544 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 15:21:56.054698 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:21:56.055000 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260925 15:21:56.056380 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:56.039090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.8.226.124:43687)
0925 15:21:56.039258 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:56.039267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:56.039312 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:21:56.039496 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:56.039501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:56.039571 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:56.039671 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:56.040612 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:56.041772 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:56.043543 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:56.044303 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:56.044428 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:56.048575 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:56.048581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:56.048586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:56.048622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:56.052396 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:56.053164 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:56.053169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:56.053173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:56.053306 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:56.053741 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:56.053746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:56.053749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:56.054118 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:56.054273 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:56.055744 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0925 15:21:56.056018 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:21:56.062111 22673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: Bootstrap starting.
I20260925 15:21:56.067514 22673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:56.068668 22673 log.cc:824] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:56.071580 22673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: No bootstrap required, opened a new log
I20260925 15:21:56.081802 22673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:56.082165 22673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:56.082269 22673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163fa78a168f44489eb40df8fad5f040, State: Initialized, Role: FOLLOWER
I20260925 15:21:56.082883 22673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:56.083839 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.084263 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:56.084987 22673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:56.109794 22761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:21:56.110080 22761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:56.111439 22761 leader_election.cc:288] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fce8a7b63cf462ea4395e5e96657ac4 (127.8.226.126:36283), 163fa78a168f44489eb40df8fad5f040 (127.8.226.125:46097)
I20260925 15:21:56.113582 22648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04bb4ea92c804ec6ae80d964d515b557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163fa78a168f44489eb40df8fad5f040" is_pre_election: true
I20260925 15:21:56.114089 22648 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04bb4ea92c804ec6ae80d964d515b557 in term 0.
I20260925 15:21:56.114535 22606 master_runner.cc:263] No leader master found from master 163fa78a168f44489eb40df8fad5f040
I20260925 15:21:56.114655 22606 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:21:56.115007 22697 leader_election.cc:302] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04bb4ea92c804ec6ae80d964d515b557, 163fa78a168f44489eb40df8fad5f040; no voters:
W20260925 15:21:56.115459 22800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:21:56.115613 22800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:21:56.115649 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04bb4ea92c804ec6ae80d964d515b557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" is_pre_election: true
I20260925 15:21:56.116165 22574 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04bb4ea92c804ec6ae80d964d515b557 in term 0.
I20260925 15:21:56.116873 22761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:21:56.117022 22761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:21:56.117115 22761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:56.120270 22761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:56.121223 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04bb4ea92c804ec6ae80d964d515b557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fce8a7b63cf462ea4395e5e96657ac4"
I20260925 15:21:56.121434 22574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:56.124698 22574 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04bb4ea92c804ec6ae80d964d515b557 in term 1.
I20260925 15:21:56.125077 22697 leader_election.cc:302] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04bb4ea92c804ec6ae80d964d515b557, 3fce8a7b63cf462ea4395e5e96657ac4; no voters:
I20260925 15:21:56.126590 22761 leader_election.cc:288] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [CANDIDATE]: Term 1 election: Requested vote from peers 3fce8a7b63cf462ea4395e5e96657ac4 (127.8.226.126:36283), 163fa78a168f44489eb40df8fad5f040 (127.8.226.125:46097)
I20260925 15:21:56.126796 22761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:56.126924 22761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [term 1 LEADER]: Becoming Leader. State: Replica: 04bb4ea92c804ec6ae80d964d515b557, State: Running, Role: LEADER
I20260925 15:21:56.127408 22761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } }
I20260925 15:21:56.145633 22648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04bb4ea92c804ec6ae80d964d515b557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163fa78a168f44489eb40df8fad5f040"
I20260925 15:21:56.145865 22648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:56.149260 22648 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04bb4ea92c804ec6ae80d964d515b557 in term 1.
I20260925 15:21:56.152056 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bb4ea92c804ec6ae80d964d515b557. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.153579 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:56.160430 22804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:56.162806 22804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:56.182307 22648 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [term 1 FOLLOWER]: Refusing update from remote peer 04bb4ea92c804ec6ae80d964d515b557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:56.183183 22802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:56.190366 22574 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [term 1 FOLLOWER]: Refusing update from remote peer 04bb4ea92c804ec6ae80d964d515b557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:56.191136 22802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:56.216099 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bb4ea92c804ec6ae80d964d515b557. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.216387 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:56.221199 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bb4ea92c804ec6ae80d964d515b557. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.221467 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:56.224134 22805 mvcc.cc:205] Tried to move back new op lower bound from 7333272437468721152 to 7333272437276483584. Current Snapshot: MvccSnapshot[applied={T|T < 7333272437468721152}]
I20260925 15:21:56.241551 22807 mvcc.cc:205] Tried to move back new op lower bound from 7333272437468721152 to 7333272437276483584. Current Snapshot: MvccSnapshot[applied={T|T < 7333272437468721152}]
I20260925 15:21:56.243849 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.244087 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:56.253423 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.253674 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:21:56.256094 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.256404 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:56.256873 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04bb4ea92c804ec6ae80d964d515b557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fce8a7b63cf462ea4395e5e96657ac4" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 36283 } } peers { permanent_uuid: "163fa78a168f44489eb40df8fad5f040" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 46097 } } peers { permanent_uuid: "04bb4ea92c804ec6ae80d964d515b557" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 43687 } } }
I20260925 15:21:56.257155 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:56.260040 22804 catalog_manager.cc:1347] Generated new cluster ID: a0574def96a24305afb0d6df33091af5
I20260925 15:21:56.260129 22804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:56.290515 22804 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:56.294428 22804 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:56.317548 22804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04bb4ea92c804ec6ae80d964d515b557: Generated new TSK 0
I20260925 15:21:56.318691 22804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:21:57.003213 22786 catalog_manager.cc:1259] Loaded cluster ID: a0574def96a24305afb0d6df33091af5
I20260925 15:21:57.003314 22786 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: loading cluster ID for follower catalog manager: success
I20260925 15:21:57.005537 22786 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: acquiring CA information for follower catalog manager: success
I20260925 15:21:57.007069 22786 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fce8a7b63cf462ea4395e5e96657ac4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:21:57.117592 22800 catalog_manager.cc:1259] Loaded cluster ID: a0574def96a24305afb0d6df33091af5
I20260925 15:21:57.117744 22800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: loading cluster ID for follower catalog manager: success
I20260925 15:21:57.120069 22800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: acquiring CA information for follower catalog manager: success
I20260925 15:21:57.122541 22800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 163fa78a168f44489eb40df8fad5f040: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:21:57.435817 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22532
I20260925 15:21:57.437626 22594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:57.545727 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22532
I20260925 15:21:57.565778 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22606
I20260925 15:21:57.566799 22668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:57.669426 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22606
I20260925 15:21:57.688303 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22680
I20260925 15:21:57.689368 22742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:57.794804 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22680
2026-09-25T15:21:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4675 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:57 dist-test-slave-8vw7 krb5kdc[22824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:57 dist-test-slave-8vw7 krb5kdc[22824](info): set up 2 sockets
Sep 25 15:21:57 dist-test-slave-8vw7 krb5kdc[22824](info): commencing operation
krb5kdc: starting...
W20260925 15:21:58.853520 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:21:58 dist-test-slave-8vw7 krb5kdc[22824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:21:59.063961 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:45207
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:45207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:59.391063 22837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:59.391311 22837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:59.391393 22837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:59.398469 22837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:59.398564 22837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:59.398607 22837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:59.398654 22837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:59.398733 22837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:59.408248 22837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:45207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:45207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:21:59.409552 22837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:59.410847 22837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:59.421906 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:59.422037 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:59.422008 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:59.423197 22837 server_base.cc:1034] running on GCE node
I20260925 15:21:59.424088 22837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:59.425572 22837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:59.426798 22837 hybrid_clock.cc:648] HybridClock initialized: now 1790349719426764 us; error 50 us; skew 500 ppm
I20260925 15:21:59.427110 22837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:21:59 dist-test-slave-8vw7 krb5kdc[22824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349719, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:59.430507 22837 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:21:59.431874 22837 webserver.cc:460] Webserver started at http://127.8.226.126:36285/ using document root <none> and password file <none>
I20260925 15:21:59.432565 22837 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:59.432691 22837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:59.432942 22837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:59.435804 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "12c48c98a410498f98c4d6b30c0fe9a9"
format_stamp: "Formatted at 2026-09-25 15:21:59 on dist-test-slave-8vw7"
server_key: "e3b219267014ac48a9da8a1612b04775"
server_key_iv: "8acce092bc1ef5502575c6cf1d906dc6"
server_key_version: "encryptionkey@0"
I20260925 15:21:59.436453 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "12c48c98a410498f98c4d6b30c0fe9a9"
format_stamp: "Formatted at 2026-09-25 15:21:59 on dist-test-slave-8vw7"
server_key: "e3b219267014ac48a9da8a1612b04775"
server_key_iv: "8acce092bc1ef5502575c6cf1d906dc6"
server_key_version: "encryptionkey@0"
I20260925 15:21:59.441252 22837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.006s
I20260925 15:21:59.444501 22852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:59.445748 22837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:21:59.445886 22837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "12c48c98a410498f98c4d6b30c0fe9a9"
format_stamp: "Formatted at 2026-09-25 15:21:59 on dist-test-slave-8vw7"
server_key: "e3b219267014ac48a9da8a1612b04775"
server_key_iv: "8acce092bc1ef5502575c6cf1d906dc6"
server_key_version: "encryptionkey@0"
I20260925 15:21:59.446033 22837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:59.459825 22837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:59.463352 22837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:59.463560 22837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:59.481068 22837 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:45207
I20260925 15:21:59.481098 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:45207 every 8 connection(s)
I20260925 15:21:59.482303 22837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:59.483537 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22837
I20260925 15:21:59.483742 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:21:59.484560 9097 external_mini_cluster.cc:1428] Setting key c998330c5a3e866283f0a03c389a6d5f
I20260925 15:21:59.486707 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:21:59 dist-test-slave-8vw7 krb5kdc[22824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:21:59.500168 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9: Bootstrap starting.
I20260925 15:21:59.503237 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:59.504164 22904 log.cc:824] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:59.506834 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9: No bootstrap required, opened a new log
I20260925 15:21:59.507370 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:59.486374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.8.226.126:45207)
0925 15:21:59.487118 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:59.487129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:59.487168 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:21:59.488748 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:59.488768 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:59.489180 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:59.489482 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:59.491297 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:59.492801 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:59.494686 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:59.495389 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:59.495510 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:59.500303 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:59.500313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:59.500329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:59.500380 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:59.503380 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:59.504131 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:59.504138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:59.504143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:59.504317 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:59.504700 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:59.504705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:59.504709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:59.505271 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:59.505572 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:59.506017 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 15:21:59.506408 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":381,"threads_started":1}
I20260925 15:21:59.510914 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:21:59.514631 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45207 } }
I20260925 15:21:59.514983 22904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:59.515089 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c48c98a410498f98c4d6b30c0fe9a9, State: Initialized, Role: FOLLOWER
I20260925 15:21:59.515668 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45207 } }
I20260925 15:21:59.515904 22904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:59.516034 22904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:59.516217 22904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:59.518344 22904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45207 } }
I20260925 15:21:59.518810 22904 leader_election.cc:302] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c48c98a410498f98c4d6b30c0fe9a9; no voters:
I20260925 15:21:59.519362 22904 leader_election.cc:288] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:21:59.519496 22909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:59.520166 22909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [term 1 LEADER]: Becoming Leader. State: Replica: 12c48c98a410498f98c4d6b30c0fe9a9, State: Running, Role: LEADER
I20260925 15:21:59.520664 22909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45207 } }
I20260925 15:21:59.521425 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:59.523309 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c48c98a410498f98c4d6b30c0fe9a9. Latest consensus state: current_term: 1 leader_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45207 } } }
I20260925 15:21:59.523284 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c48c98a410498f98c4d6b30c0fe9a9" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45207 } } }
I20260925 15:21:59.523711 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:59.523715 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:59.525965 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:59.531183 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:59.539006 22914 catalog_manager.cc:1347] Generated new cluster ID: 5aaf4b8b425b4d088921757dedc7cbb0
I20260925 15:21:59.539100 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:59.558542 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:59.560366 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:59.576064 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c48c98a410498f98c4d6b30c0fe9a9: Generated new TSK 0
I20260925 15:21:59.576807 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:21:59.622663 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22837
I20260925 15:21:59.623812 22899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:21:59.716706 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22837
2026-09-25T15:21:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1917 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:59 dist-test-slave-8vw7 krb5kdc[22932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:59 dist-test-slave-8vw7 krb5kdc[22932](info): set up 2 sockets
Sep 25 15:21:59 dist-test-slave-8vw7 krb5kdc[22932](info): commencing operation
krb5kdc: starting...
W20260925 15:22:00.781492 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:22:00 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:22:00.991734 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34559
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:34559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:01.308449 22945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:01.308699 22945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:01.308801 22945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:01.315760 22945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:01.315871 22945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:01.315914 22945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:01.315995 22945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:01.316082 22945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:01.325728 22945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:34559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:01.326963 22945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:01.328245 22945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:01.339754 22950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:01.339928 22951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:01.339958 22953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:01.340797 22945 server_base.cc:1034] running on GCE node
I20260925 15:22:01.341530 22945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:01.342809 22945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:01.344034 22945 hybrid_clock.cc:648] HybridClock initialized: now 1790349721343991 us; error 55 us; skew 500 ppm
I20260925 15:22:01.344354 22945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:01 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:01.350785 22945 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:22:01.352217 22945 webserver.cc:460] Webserver started at http://127.8.226.126:45119/ using document root <none> and password file <none>
I20260925 15:22:01.352842 22945 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:01.353006 22945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:01.353314 22945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:01.356123 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "f9bfa36fdb864f889eb268b754e731cd"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "34e66099619369576c1f238693baa4c4"
server_key_iv: "c6a7058c927db5141f264c5aee232459"
server_key_version: "encryptionkey@0"
I20260925 15:22:01.356863 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "f9bfa36fdb864f889eb268b754e731cd"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "34e66099619369576c1f238693baa4c4"
server_key_iv: "c6a7058c927db5141f264c5aee232459"
server_key_version: "encryptionkey@0"
I20260925 15:22:01.361644 22945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:22:01.364787 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:01.365983 22945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:22:01.366159 22945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "f9bfa36fdb864f889eb268b754e731cd"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "34e66099619369576c1f238693baa4c4"
server_key_iv: "c6a7058c927db5141f264c5aee232459"
server_key_version: "encryptionkey@0"
I20260925 15:22:01.366338 22945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:01.382628 22945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:01.386339 22945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:01.386598 22945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:01.404770 22945 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:34559
I20260925 15:22:01.404839 23011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:34559 every 8 connection(s)
I20260925 15:22:01.406167 22945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:01.409713 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:01.412122 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 22945
I20260925 15:22:01.412343 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:22:01.413237 9097 external_mini_cluster.cc:1428] Setting key 1ecc4ab34bb9437d463509acb9908eee
I20260925 15:22:01.421948 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap starting.
I20260925 15:22:01.425530 23012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:01.426831 23012 log.cc:824] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:22:01 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:01.429910 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: No bootstrap required, opened a new log
I20260925 15:22:01.437090 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:01.415234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.8.226.126:34559)
0925 15:22:01.415997 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:01.416007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:01.416044 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:22:01.417754 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:01.417775 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:01.418142 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:01.418466 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:01.420097 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:01.421450 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:01.423313 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:01.423975 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:01.424114 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:01.429034 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:01.429048 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:01.429067 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:01.429129 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:01.432983 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:01.433599 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:01.433606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:01.433614 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:01.433811 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:01.434132 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:01.434136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:01.434140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:01.434796 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:01.435093 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:01.435359 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 15:22:01.435802 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":368,"threads_started":1}
I20260925 15:22:01.440220 23012 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:01.440518 23012 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:01.440587 23012 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bfa36fdb864f889eb268b754e731cd, State: Initialized, Role: FOLLOWER
I20260925 15:22:01.441077 23012 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:01.441269 23012 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:01.441362 23012 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:01.441473 23012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:01.443476 23012 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:01.444007 23012 leader_election.cc:302] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9bfa36fdb864f889eb268b754e731cd; no voters:
I20260925 15:22:01.444567 23012 leader_election.cc:288] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:22:01.444686 23017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:01.445367 23017 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 LEADER]: Becoming Leader. State: Replica: f9bfa36fdb864f889eb268b754e731cd, State: Running, Role: LEADER
I20260925 15:22:01.445895 23017 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:01.446517 23012 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:01.448559 23018 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9bfa36fdb864f889eb268b754e731cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } } }
I20260925 15:22:01.448884 23018 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:01.449703 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bfa36fdb864f889eb268b754e731cd. Latest consensus state: current_term: 1 leader_uuid: "f9bfa36fdb864f889eb268b754e731cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } } }
I20260925 15:22:01.450103 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:01.451422 23024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:01.456478 23024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:01.465037 23024 catalog_manager.cc:1347] Generated new cluster ID: bfe910b5bdc34efdb32647d81f9e9cae
I20260925 15:22:01.465145 23024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:01.481711 23024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:22:01.483283 23024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:01.495208 23024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Generated new TSK 0
I20260925 15:22:01.496078 23024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:22:01.554195 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34559
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:01.862736 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:01.862970 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:01.863114 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:01.869771 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:01.869880 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:01.870054 23038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:22:01.878932 23038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:01.880012 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:01.881230 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:01.893388 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:01.893429 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:01.893499 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:01.894253 23038 server_base.cc:1034] running on GCE node
I20260925 15:22:01.894833 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:01.895654 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:01.896889 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790349721896853 us; error 30 us; skew 500 ppm
I20260925 15:22:01.897207 23038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:01 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:01.900152 23038 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:22:01.901526 23038 webserver.cc:460] Webserver started at http://127.8.226.65:41341/ using document root <none> and password file <none>
I20260925 15:22:01.902164 23038 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:01.902300 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:01.902577 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:01.905449 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "4418058d4a094d56976003b0966ec38a"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "67f6916f762e1693409170452aba0fd6"
server_key_iv: "6855f78ea53ae7ae182651c75ad4d392"
server_key_version: "encryptionkey@0"
I20260925 15:22:01.906225 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "4418058d4a094d56976003b0966ec38a"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "67f6916f762e1693409170452aba0fd6"
server_key_iv: "6855f78ea53ae7ae182651c75ad4d392"
server_key_version: "encryptionkey@0"
I20260925 15:22:01.910961 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:22:01.914132 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:01.915333 23038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:01.915498 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "4418058d4a094d56976003b0966ec38a"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "67f6916f762e1693409170452aba0fd6"
server_key_iv: "6855f78ea53ae7ae182651c75ad4d392"
server_key_version: "encryptionkey@0"
I20260925 15:22:01.915681 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:01.943017 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:01.946822 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:01.947081 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:01.948072 23038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:01.949299 23038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:01.949393 23038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:01.949529 23038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:01.949608 23038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:01.981380 23038 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43423
I20260925 15:22:01.981427 23165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43423 every 8 connection(s)
I20260925 15:22:01.982576 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:01.984380 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23038
I20260925 15:22:01.984615 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:22:01.985478 9097 external_mini_cluster.cc:1428] Setting key 4ddcbb455c043cb96abb5a6f009025fc
Sep 25 15:22:01 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:02.003949 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:01.985103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:32969 (local address 127.8.226.126:34559)
0925 15:22:01.985563 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:01.985571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:01.986732 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 15:22:01.988479 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:01.988484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:01.988574 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:01.988684 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:01.990197 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:01.990937 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:01.992539 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:01.992917 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:01.993106 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:01.997304 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:01.997312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:01.997319 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:01.997395 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:02.000771 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.001475 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.001482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.001488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.001634 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.002005 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.002012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.002018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.002419 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:02.002577 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:02.003210 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 15:22:02.003486 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":245,"threads_started":1}
I20260925 15:22:02.004392 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:01.985803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:32969)
0925 15:22:01.986580 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:01.986630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:01.988248 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:01.988891 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:01.988899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:01.989425 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:01.990020 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:01.990036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:01.991106 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:01.991110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:01.992377 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:01.992389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:01.993190 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:01.993212 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:01.993447 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:01.994228 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:01.994261 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:01.997096 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:02.000887 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:02.000892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:02.000902 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:02.001326 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:02.001737 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:02.001739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:02.001741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:02.001884 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:02.002528 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:02.002533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:02.002836 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 15:22:02.003035 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:22:02.003375 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260925 15:22:02.005892 23166 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34559
I20260925 15:22:02.006227 23166 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:02.007076 23166 heartbeater.cc:511] Master 127.8.226.126:34559 requested a full tablet report, sending...
I20260925 15:22:02.009299 22977 ts_manager.cc:194] Registered new tserver with Master: 4418058d4a094d56976003b0966ec38a (127.8.226.65:43423)
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:22:02.010766 22977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:32969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:22:02.037109 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34559
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:02.340191 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:02.340422 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:02.340512 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:02.347416 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:02.347517 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:02.347697 23172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:22:02.356626 23172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:02.357729 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:02.358902 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:02.371106 23177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:02.371145 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:02.371193 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:02.371970 23172 server_base.cc:1034] running on GCE node
I20260925 15:22:02.372459 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:02.373426 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:02.374603 23172 hybrid_clock.cc:648] HybridClock initialized: now 1790349722374563 us; error 59 us; skew 500 ppm
I20260925 15:22:02.374893 23172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:02 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349722, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:02.377820 23172 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:22:02.379038 23172 webserver.cc:460] Webserver started at http://127.8.226.66:41847/ using document root <none> and password file <none>
I20260925 15:22:02.379666 23172 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:02.379801 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:02.380074 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:02.382722 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "573a89add19848d7bc80c02ef23bcdfe"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "a3a3c6f7ed7a917f94a5770dc8b3dc8e"
server_key_iv: "2de3cea861f9f4c65972bcc2bd7ec7ce"
server_key_version: "encryptionkey@0"
I20260925 15:22:02.383424 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "573a89add19848d7bc80c02ef23bcdfe"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "a3a3c6f7ed7a917f94a5770dc8b3dc8e"
server_key_iv: "2de3cea861f9f4c65972bcc2bd7ec7ce"
server_key_version: "encryptionkey@0"
I20260925 15:22:02.388003 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:22:02.391158 23187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:02.392298 23172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:02.392460 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "573a89add19848d7bc80c02ef23bcdfe"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "a3a3c6f7ed7a917f94a5770dc8b3dc8e"
server_key_iv: "2de3cea861f9f4c65972bcc2bd7ec7ce"
server_key_version: "encryptionkey@0"
I20260925 15:22:02.392634 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:02.418594 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:02.422736 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:02.423009 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:02.423827 23172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:02.425109 23172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:02.425210 23172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:02.425350 23172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:02.425417 23172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:02.456615 23172 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:41901
I20260925 15:22:02.456655 23299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:41901 every 8 connection(s)
I20260925 15:22:02.457837 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:02.466529 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23172
I20260925 15:22:02.466724 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:22:02.467162 9097 external_mini_cluster.cc:1428] Setting key 8989ecddc750bb55be8f5d27e299f6a4
Sep 25 15:22:02 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349722, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:02.477332 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:02.460165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:34879 (local address 127.8.226.126:34559)
0925 15:22:02.460338 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:02.460342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:02.461421 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 15:22:02.463260 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:02.463266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:02.463350 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:02.463457 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:02.464827 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.465622 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.467268 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.467652 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.467784 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:02.471594 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:02.471598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:02.471601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:02.471627 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:02.474094 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.474953 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.474956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.474959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.475074 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.475536 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.475539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.475541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.475771 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:02.475873 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:02.476674 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 15:22:02.476915 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:22:02.477928 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:02.460599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:34879)
0925 15:22:02.461279 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:02.461314 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:02.463006 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:02.463672 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:02.463680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:02.464147 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:02.464670 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:02.464686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.465827 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.465832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:02.467130 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:02.467141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.467873 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.467879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:02.468100 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:02.468740 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:02.468768 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:02.471416 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:02.474226 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:02.474237 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:02.474253 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:02.474849 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:02.475189 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:02.475196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:02.475202 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:02.475433 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:02.475887 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:02.475897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:02.476332 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 15:22:02.476536 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 15:22:02.476868 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":288,"threads_started":1}
I20260925 15:22:02.479166 23300 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34559
I20260925 15:22:02.479450 23300 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:02.480080 23300 heartbeater.cc:511] Master 127.8.226.126:34559 requested a full tablet report, sending...
I20260925 15:22:02.481400 22977 ts_manager.cc:194] Registered new tserver with Master: 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901)
I20260925 15:22:02.482117 22977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:34879
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:22:02.515628 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34559
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:02.823812 23305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:02.824113 23305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:02.824260 23305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:02.830984 23305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:02.831122 23305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:02.831285 23305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:22:02.840191 23305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:02.841455 23305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:02.842707 23305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:02.855154 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:02.855224 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:02.855248 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:02.856112 23305 server_base.cc:1034] running on GCE node
I20260925 15:22:02.856683 23305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:02.857630 23305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:02.858860 23305 hybrid_clock.cc:648] HybridClock initialized: now 1790349722858814 us; error 37 us; skew 500 ppm
I20260925 15:22:02.859153 23305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:02 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349722, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:02.862171 23305 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:22:02.863487 23305 webserver.cc:460] Webserver started at http://127.8.226.67:36549/ using document root <none> and password file <none>
I20260925 15:22:02.864187 23305 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:02.864326 23305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:02.864606 23305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:02.867410 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "df3b2f05be1a4b8baa671f8f5fff2701"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "42564ba0a58de91e683fb6e82d8c5dab"
server_key_iv: "c94ad7e361e211f42c0670fdac1631d1"
server_key_version: "encryptionkey@0"
I20260925 15:22:02.868184 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "df3b2f05be1a4b8baa671f8f5fff2701"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "42564ba0a58de91e683fb6e82d8c5dab"
server_key_iv: "c94ad7e361e211f42c0670fdac1631d1"
server_key_version: "encryptionkey@0"
I20260925 15:22:02.873044 23305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:22:02.876089 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:02.877286 23305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:22:02.877445 23305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "df3b2f05be1a4b8baa671f8f5fff2701"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "42564ba0a58de91e683fb6e82d8c5dab"
server_key_iv: "c94ad7e361e211f42c0670fdac1631d1"
server_key_version: "encryptionkey@0"
I20260925 15:22:02.877638 23305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:02.908074 23305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:02.912042 23305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:02.912322 23305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:02.913264 23305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:02.914534 23305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:02.914635 23305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:02.914789 23305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:02.914861 23305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:02.946962 23305 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40363
I20260925 15:22:02.947067 23432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40363 every 8 connection(s)
I20260925 15:22:02.948249 23305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:02.954097 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23305
I20260925 15:22:02.954396 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:22:02.954811 9097 external_mini_cluster.cc:1428] Setting key 687c618a8fa7c33442159cc207a67781
Sep 25 15:22:02 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349722, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:02.967224 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:02.950747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:43991 (local address 127.8.226.126:34559)
0925 15:22:02.950920 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:02.950924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:02.952030 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 15:22:02.953928 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:02.953934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:02.954028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:02.954142 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:02.955766 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.956786 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.958407 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.958811 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.958904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:02.962186 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:02.962193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:02.962196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:02.962226 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:02.964737 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.965384 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.965388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.965390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.965496 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.965827 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.965830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.965831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.966059 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:02.966161 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:02.966856 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 15:22:02.967002 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:22:02.968040 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:02.951160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:43991)
0925 15:22:02.951845 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:02.951911 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:02.953665 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:02.954412 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:02.954420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:02.954967 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:02.955548 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:02.955576 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.956943 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.956947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:02.958261 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:02.958273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.958979 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.958983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:02.959171 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:02.959756 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:02.959780 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:02.962018 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:02.964840 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:02.964845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:02.964856 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:02.965287 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:02.965579 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:02.965582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:02.965584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:02.965735 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:02.966176 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:02.966185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:02.966545 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 15:22:02.966766 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 15:22:02.967066 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260925 15:22:02.969269 23433 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34559
I20260925 15:22:02.969558 23433 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:02.970324 23433 heartbeater.cc:511] Master 127.8.226.126:34559 requested a full tablet report, sending...
I20260925 15:22:02.971575 22977 ts_manager.cc:194] Registered new tserver with Master: df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
I20260925 15:22:02.972329 22977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:43991
I20260925 15:22:02.982134 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:02.998227 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:02.986553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.8.226.126:34559)
0925 15:22:02.986749 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:02.986757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:02.987097 (+ 340us) server_negotiation.cc:407] Connection header received
0925 15:22:02.987289 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:02.987294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:02.987369 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:02.987467 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:02.988757 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.989447 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.990754 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:02.991127 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:02.991254 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:02.993527 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:02.993531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:02.993534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:02.993559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:02.995610 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.996270 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.996276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.996280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.996407 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:02.996800 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:02.996807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:02.996813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:02.997205 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:02.997448 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:02.997767 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 15:22:02.997955 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:22:03.001931 22977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:03.008675 22977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:03.013852 23166 heartbeater.cc:503] Master 127.8.226.126:34559 was elected leader, sending a full tablet report...
I20260925 15:22:03.045888 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.021164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.8.226.66:41901)
0925 15:22:03.032364 (+ 11200us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.032375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.033094 (+ 719us) server_negotiation.cc:407] Connection header received
0925 15:22:03.033241 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.033252 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.033566 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.033788 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:03.034590 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.038672 (+ 4082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.043228 (+ 4556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.044615 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.044761 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.044974 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.045380 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.045545 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11037,"thread_start_us":10884,"threads_started":1}
I20260925 15:22:03.045918 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.020874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41901 (local address 127.0.0.1:43710)
0925 15:22:03.032711 (+ 11837us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.032732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:03.032881 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:03.033832 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:03.033843 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:03.033884 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:03.034375 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.034390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.038868 (+ 4478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.038876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.042895 (+ 4019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.042938 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.044840 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.044846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.045144 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.045203 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:22:03.045324 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:22:03.045508 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11683,"mutex_wait_us":53,"thread_start_us":11513,"threads_started":1}
I20260925 15:22:03.047034 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.020316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44892)
0925 15:22:03.031238 (+ 10922us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.031295 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:03.031503 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:03.039275 (+ 7772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:03.039293 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:03.039318 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:03.039760 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.039771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.041783 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.041788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.044194 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.044209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.046297 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.046303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.046455 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.046488 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:22:03.046585 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:22:03.046714 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10597,"thread_start_us":10457,"threads_started":1}
I20260925 15:22:03.047102 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.020437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.8.226.65:43423)
0925 15:22:03.038520 (+ 18083us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.038527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.038553 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:22:03.038645 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.038651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.038857 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.039038 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:03.039999 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.041636 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.044373 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.045769 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.045899 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.046110 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.046645 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.046812 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17934,"thread_start_us":18552,"threads_started":1}
I20260925 15:22:03.048525 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.031751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40363 (local address 127.0.0.1:55882)
0925 15:22:03.036202 (+ 4451us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.036221 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:03.036332 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:03.039760 (+ 3428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:03.039765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:03.039807 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:03.040281 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.040293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.042198 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.042203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.045147 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.045159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.047221 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.047229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.047555 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.047577 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:22:03.047744 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:22:03.047898 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4307,"mutex_wait_us":3,"thread_start_us":4060,"threads_started":1}
I20260925 15:22:03.049453 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.032147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.8.226.67:40363)
0925 15:22:03.038157 (+ 6010us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.038169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.038201 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:22:03.038294 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.038300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.038524 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.039337 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:03.040506 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.042018 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.045319 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.046746 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.046866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.047077 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.049062 (+ 1985us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.049219 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5823,"thread_start_us":5659,"threads_started":1}
I20260925 15:22:03.050994 23235 tablet_service.cc:1467] Processing CreateTablet for tablet 171804c6c23a4b58b11a4e2ff758f46b (DEFAULT_TABLE table=test-table [id=defa3edb10a749c89cd7e18f33275219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:03.051616 23101 tablet_service.cc:1467] Processing CreateTablet for tablet 171804c6c23a4b58b11a4e2ff758f46b (DEFAULT_TABLE table=test-table [id=defa3edb10a749c89cd7e18f33275219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:03.052582 23235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171804c6c23a4b58b11a4e2ff758f46b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:03.053033 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171804c6c23a4b58b11a4e2ff758f46b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:03.054385 23368 tablet_service.cc:1467] Processing CreateTablet for tablet 171804c6c23a4b58b11a4e2ff758f46b (DEFAULT_TABLE table=test-table [id=defa3edb10a749c89cd7e18f33275219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:03.055536 23368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171804c6c23a4b58b11a4e2ff758f46b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:03.065639 23451 tablet_bootstrap.cc:492] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Bootstrap starting.
I20260925 15:22:03.067780 23453 tablet_bootstrap.cc:492] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Bootstrap starting.
I20260925 15:22:03.069106 23451 tablet_bootstrap.cc:654] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:03.070681 23451 log.cc:824] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:03.071496 23453 tablet_bootstrap.cc:654] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:03.072736 23453 log.cc:824] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:03.073107 23452 tablet_bootstrap.cc:492] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Bootstrap starting.
I20260925 15:22:03.074054 23451 tablet_bootstrap.cc:492] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: No bootstrap required, opened a new log
I20260925 15:22:03.074385 23451 ts_tablet_manager.cc:1397] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260925 15:22:03.075609 23452 tablet_bootstrap.cc:654] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:03.076508 23452 log.cc:824] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:03.079144 23453 tablet_bootstrap.cc:492] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: No bootstrap required, opened a new log
I20260925 15:22:03.079385 23453 ts_tablet_manager.cc:1397] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260925 15:22:03.079430 23452 tablet_bootstrap.cc:492] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: No bootstrap required, opened a new log
I20260925 15:22:03.079641 23452 ts_tablet_manager.cc:1397] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260925 15:22:03.083957 23451 raft_consensus.cc:348] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.084295 23451 raft_consensus.cc:374] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:03.084412 23451 raft_consensus.cc:729] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b2f05be1a4b8baa671f8f5fff2701, State: Initialized, Role: FOLLOWER
I20260925 15:22:03.085022 23451 consensus_queue.cc:260] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.086025 23433 heartbeater.cc:503] Master 127.8.226.126:34559 was elected leader, sending a full tablet report...
I20260925 15:22:03.086663 23451 ts_tablet_manager.cc:1428] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 15:22:03.089875 23452 raft_consensus.cc:348] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.089872 23453 raft_consensus.cc:348] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.090294 23453 raft_consensus.cc:374] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:03.090353 23452 raft_consensus.cc:374] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:03.090451 23453 raft_consensus.cc:729] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4418058d4a094d56976003b0966ec38a, State: Initialized, Role: FOLLOWER
I20260925 15:22:03.090451 23452 raft_consensus.cc:729] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573a89add19848d7bc80c02ef23bcdfe, State: Initialized, Role: FOLLOWER
I20260925 15:22:03.091018 23453 consensus_queue.cc:260] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.091019 23452 consensus_queue.cc:260] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.092594 23300 heartbeater.cc:503] Master 127.8.226.126:34559 was elected leader, sending a full tablet report...
I20260925 15:22:03.092759 23453 ts_tablet_manager.cc:1428] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:22:03.093125 23452 ts_tablet_manager.cc:1428] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
W20260925 15:22:03.199817 23434 tablet.cc:2369] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:22:03.209664 23301 tablet.cc:2369] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:03.228529 23457 raft_consensus.cc:482] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:03.228820 23457 raft_consensus.cc:504] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.230641 23457 leader_election.cc:288] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4418058d4a094d56976003b0966ec38a (127.8.226.65:43423), 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901)
W20260925 15:22:03.234782 23167 tablet.cc:2369] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:03.237429 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.230852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.8.226.67:52817)
0925 15:22:03.231159 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.231193 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:03.231323 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:03.231761 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:03.231766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:03.231794 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:03.232008 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.232021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.233346 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.233350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.235363 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.235378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.236748 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.236752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.236876 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.236900 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:22:03.237037 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:22:03.237138 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":6}
I20260925 15:22:03.237455 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.231065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:52817 (local address 127.8.226.65:43423)
0925 15:22:03.231286 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.231294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.231316 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:22:03.231495 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.231503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.231617 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.231718 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:03.232180 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.233226 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.235535 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.236606 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.236773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.236848 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.237069 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.237204 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:22:03.238194 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.231165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:59311 (local address 127.8.226.66:41901)
0925 15:22:03.231350 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.231356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.232038 (+ 682us) server_negotiation.cc:407] Connection header received
0925 15:22:03.232196 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.232201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.232289 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.232393 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:03.232856 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.233731 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.235949 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.237191 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.237335 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.237397 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.237808 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.237933 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:22:03.238262 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.231149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41901 (local address 127.8.226.67:59311)
0925 15:22:03.231898 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.231919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:03.232035 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:03.232407 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:03.232410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:03.232428 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:03.232705 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.232712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.233890 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.233895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.235810 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:03.235820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.237385 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.237390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:03.237578 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.237619 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:22:03.237780 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 15:22:03.237899 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":28,"thread_start_us":415,"threads_started":1}
I20260925 15:22:03.238395 23121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171804c6c23a4b58b11a4e2ff758f46b" candidate_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4418058d4a094d56976003b0966ec38a" is_pre_election: true
I20260925 15:22:03.238909 23121 raft_consensus.cc:2454] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3b2f05be1a4b8baa671f8f5fff2701 in term 0.
I20260925 15:22:03.239303 23255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171804c6c23a4b58b11a4e2ff758f46b" candidate_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573a89add19848d7bc80c02ef23bcdfe" is_pre_election: true
I20260925 15:22:03.239606 23324 leader_election.cc:302] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4418058d4a094d56976003b0966ec38a, df3b2f05be1a4b8baa671f8f5fff2701; no voters:
I20260925 15:22:03.239872 23255 raft_consensus.cc:2454] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3b2f05be1a4b8baa671f8f5fff2701 in term 0.
I20260925 15:22:03.240031 23457 raft_consensus.cc:2781] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:03.240185 23457 raft_consensus.cc:482] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:03.240294 23457 raft_consensus.cc:3037] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:03.242583 23457 raft_consensus.cc:504] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.243139 23457 leader_election.cc:288] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [CANDIDATE]: Term 1 election: Requested vote from peers 4418058d4a094d56976003b0966ec38a (127.8.226.65:43423), 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901)
I20260925 15:22:03.243520 23121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171804c6c23a4b58b11a4e2ff758f46b" candidate_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4418058d4a094d56976003b0966ec38a"
I20260925 15:22:03.243521 23255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171804c6c23a4b58b11a4e2ff758f46b" candidate_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573a89add19848d7bc80c02ef23bcdfe"
I20260925 15:22:03.243724 23121 raft_consensus.cc:3037] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:03.243713 23255 raft_consensus.cc:3037] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:03.246871 23255 raft_consensus.cc:2454] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3b2f05be1a4b8baa671f8f5fff2701 in term 1.
I20260925 15:22:03.246869 23121 raft_consensus.cc:2454] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3b2f05be1a4b8baa671f8f5fff2701 in term 1.
I20260925 15:22:03.247251 23324 leader_election.cc:302] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4418058d4a094d56976003b0966ec38a, df3b2f05be1a4b8baa671f8f5fff2701; no voters:
I20260925 15:22:03.247545 23457 raft_consensus.cc:2781] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:03.248126 23457 raft_consensus.cc:686] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 LEADER]: Becoming Leader. State: Replica: df3b2f05be1a4b8baa671f8f5fff2701, State: Running, Role: LEADER
I20260925 15:22:03.248582 23457 consensus_queue.cc:237] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } }
I20260925 15:22:03.253347 22976 catalog_manager.cc:5495] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67). New cstate: current_term: 1 leader_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } health_report { overall_health: UNKNOWN } } }
I20260925 15:22:03.279971 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.274961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.8.226.67:40363)
0925 15:22:03.275194 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.275198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.275215 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:22:03.275364 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.275367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.275450 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.275566 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:03.275906 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.276611 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.277947 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.278370 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.278460 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.278886 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:03.279381 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:03.279494 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.279585 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.279649 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:22:03.287765 23121 raft_consensus.cc:1261] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Refusing update from remote peer df3b2f05be1a4b8baa671f8f5fff2701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:03.287779 23255 raft_consensus.cc:1261] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Refusing update from remote peer df3b2f05be1a4b8baa671f8f5fff2701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:03.288682 23457 consensus_queue.cc:1035] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:03.288990 23461 consensus_queue.cc:1035] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:03.306313 23465 mvcc.cc:205] Tried to move back new op lower bound from 7333272466576408576 to 7333272466429792256. Current Snapshot: MvccSnapshot[applied={T|T < 7333272466576408576}]
I20260925 15:22:03.313866 22976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:03.314155 22976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:03.317744 22976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Sending DeleteTablet for 3 replicas of tablet 171804c6c23a4b58b11a4e2ff758f46b
I20260925 15:22:03.318958 23101 tablet_service.cc:1514] Processing DeleteTablet for tablet 171804c6c23a4b58b11a4e2ff758f46b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:03 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:44892
I20260925 15:22:03.318982 23368 tablet_service.cc:1514] Processing DeleteTablet for tablet 171804c6c23a4b58b11a4e2ff758f46b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:03 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:55882
I20260925 15:22:03.318982 23235 tablet_service.cc:1514] Processing DeleteTablet for tablet 171804c6c23a4b58b11a4e2ff758f46b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:03 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:43710
I20260925 15:22:03.320082 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23038
I20260925 15:22:03.345521 23476 tablet_replica.cc:330] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: stopping tablet replica
I20260925 15:22:03.345696 23475 tablet_replica.cc:330] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: stopping tablet replica
I20260925 15:22:03.346143 23476 raft_consensus.cc:2229] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:22:03.346129 23475 raft_consensus.cc:2229] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:03.346696 23475 raft_consensus.cc:2258] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:03.346992 23476 raft_consensus.cc:2258] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:03.350221 23476 ts_tablet_manager.cc:1905] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:03.353654 23477 tablet_replica.cc:330] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: stopping tablet replica
I20260925 15:22:03.354099 23477 raft_consensus.cc:2229] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:03.354697 23477 raft_consensus.cc:2258] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:03.357046 23475 ts_tablet_manager.cc:1905] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:22:03 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:22:03.356918 23477 ts_tablet_manager.cc:1905] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:03.360551 23476 ts_tablet_manager.cc:1918] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:03.360770 23476 log.cc:1196] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/171804c6c23a4b58b11a4e2ff758f46b
I20260925 15:22:03.361354 23476 ts_tablet_manager.cc:1939] T 171804c6c23a4b58b11a4e2ff758f46b P df3b2f05be1a4b8baa671f8f5fff2701: Deleting consensus metadata
I20260925 15:22:03.363204 22964 catalog_manager.cc:4841] TS df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): tablet 171804c6c23a4b58b11a4e2ff758f46b (table test-table [id=defa3edb10a749c89cd7e18f33275219]) successfully deleted
I20260925 15:22:03.366416 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.322891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.8.226.65:43423)
0925 15:22:03.323134 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.323145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.344443 (+ 21298us) server_negotiation.cc:407] Connection header received
0925 15:22:03.344736 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.344745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.344931 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.345076 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:03.348173 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.349348 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.351403 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.352156 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.352253 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.358575 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:03.358588 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:03.358605 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:03.358657 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:03.362653 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:03.363639 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:03.363648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:03.363651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:03.363823 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:03.364269 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:03.364293 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:03.364297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:03.364763 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:03.365074 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.365736 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.365959 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:22:03.366458 23477 ts_tablet_manager.cc:1918] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:03.366618 23477 log.cc:1196] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/171804c6c23a4b58b11a4e2ff758f46b
I20260925 15:22:03.367180 23477 ts_tablet_manager.cc:1939] T 171804c6c23a4b58b11a4e2ff758f46b P 573a89add19848d7bc80c02ef23bcdfe: Deleting consensus metadata
I20260925 15:22:03.368249 23161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:03.369109 22963 catalog_manager.cc:4841] TS 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): tablet 171804c6c23a4b58b11a4e2ff758f46b (table test-table [id=defa3edb10a749c89cd7e18f33275219]) successfully deleted
I20260925 15:22:03.494684 23475 ts_tablet_manager.cc:1918] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:03.494930 23475 log.cc:1196] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/171804c6c23a4b58b11a4e2ff758f46b
I20260925 15:22:03.495450 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23038
I20260925 15:22:03.495524 23475 ts_tablet_manager.cc:1939] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Deleting consensus metadata
I20260925 15:22:03.512609 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23172
W20260925 15:22:03.512588 22964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:43423 (error 108)
I20260925 15:22:03.514420 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.513941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44906)
0925 15:22:03.514128 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.514252 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:22:03.514685 22964 catalog_manager.cc:4584] TS 4418058d4a094d56976003b0966ec38a (127.8.226.65:43423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 171804c6c23a4b58b11a4e2ff758f46b: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Sep 25 15:22:03 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:22:03.528047 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.513441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.8.226.66:41901)
0925 15:22:03.513805 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.513814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.513836 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:22:03.513947 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.513952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.514053 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.514189 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:03.515600 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.516373 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.517819 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.518200 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.518323 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.522447 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:03.522459 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:03.522472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:03.522503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:03.525413 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:03.526057 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:03.526064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:03.526066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:03.526188 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:03.526553 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:03.526559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:03.526562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:03.526911 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:03.527230 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.527502 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.527718 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260925 15:22:03.528652 23295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:03.568261 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.567756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44912)
0925 15:22:03.567968 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.568098 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:22:03.569447 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.569070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44914)
0925 15:22:03.569227 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.569306 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:22:03.619247 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23172
I20260925 15:22:03.630296 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.629707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44924)
0925 15:22:03.629962 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.630078 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 15:22:03.631477 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.631077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44926)
0925 15:22:03.631248 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.631341 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:22:03.636631 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23305
Sep 25 15:22:03 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:22:03.654749 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.637473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.8.226.67:40363)
0925 15:22:03.637670 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:03.637677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:03.637699 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:22:03.637768 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:03.637775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:03.637868 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:03.637994 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:03.639311 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.640630 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.642236 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:03.642896 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:03.643038 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:03.647342 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:03.647357 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:03.647377 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:03.647416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:03.651691 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:03.652388 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:03.652399 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:03.652405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:03.652579 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:03.652991 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:03.652996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:03.653000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:03.653486 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:03.653828 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:03.654110 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:22:03.654338 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:22:03.655483 23428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:03.712073 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.711605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44940)
0925 15:22:03.711817 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.711906 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:22:03.713145 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.712718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44956)
0925 15:22:03.712923 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.713040 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 15:22:03.778409 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23305
I20260925 15:22:03.797712 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 22945
I20260925 15:22:03.798856 23007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:03.906733 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:03.906056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:44958)
0925 15:22:03.906432 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:03.906530 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260925 15:22:03.906759 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 22945
I20260925 15:22:03.926887 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34559
--webserver_interface=127.8.226.126
--webserver_port=45119
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:34559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:04.240242 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:04.240501 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:04.240584 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:04.247394 23484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:04.247486 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:04.247538 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:04.247592 23484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:04.247633 23484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:04.256922 23484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:34559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=45119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:04.258113 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:04.259269 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:04.280807 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:04.280840 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:04.280807 23493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:04.281728 23484 server_base.cc:1034] running on GCE node
I20260925 15:22:04.282542 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:04.284183 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:04.285387 23484 hybrid_clock.cc:648] HybridClock initialized: now 1790349724285322 us; error 52 us; skew 500 ppm
I20260925 15:22:04.285789 23484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:04 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:04.289327 23484 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:22:04.290948 23484 webserver.cc:460] Webserver started at http://127.8.226.126:45119/ using document root <none> and password file <none>
I20260925 15:22:04.291664 23484 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:04.291810 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:04.297384 23484 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:22:04.300218 23500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:04.301491 23484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:04.301666 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "f9bfa36fdb864f889eb268b754e731cd"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "34e66099619369576c1f238693baa4c4"
server_key_iv: "c6a7058c927db5141f264c5aee232459"
server_key_version: "encryptionkey@0"
I20260925 15:22:04.302673 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:04.324294 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:04.328280 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:04.328537 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:04.346844 23484 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:34559
I20260925 15:22:04.346851 23551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:34559 every 8 connection(s)
I20260925 15:22:04.348282 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:04.354785 23552 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:22:04.357569 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23484
I20260925 15:22:04.357964 23552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap starting.
I20260925 15:22:04.358381 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:43423
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:34559
--webserver_port=41341
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:04.376803 23552 log.cc:824] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:04.388597 23552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:04.389192 23552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap complete.
I20260925 15:22:04.398427 23552 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:04.399276 23552 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bfa36fdb864f889eb268b754e731cd, State: Initialized, Role: FOLLOWER
I20260925 15:22:04.399986 23552 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:04.400236 23552 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:04.400353 23552 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:04.400530 23552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:22:04.404203 23552 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:04.404722 23552 leader_election.cc:302] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9bfa36fdb864f889eb268b754e731cd; no voters:
I20260925 15:22:04.405436 23552 leader_election.cc:288] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:22:04.405562 23556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:22:04.407239 23556 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 LEADER]: Becoming Leader. State: Replica: f9bfa36fdb864f889eb268b754e731cd, State: Running, Role: LEADER
I20260925 15:22:04.407841 23556 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:04.408105 23552 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:04.410363 23558 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bfa36fdb864f889eb268b754e731cd. Latest consensus state: current_term: 2 leader_uuid: "f9bfa36fdb864f889eb268b754e731cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } } }
I20260925 15:22:04.411309 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f9bfa36fdb864f889eb268b754e731cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } } }
I20260925 15:22:04.411646 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:04.412122 23558 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:04.414620 23562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:04.423959 23562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:04.425832 23562 catalog_manager.cc:1259] Loaded cluster ID: bfe910b5bdc34efdb32647d81f9e9cae
I20260925 15:22:04.426002 23562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:04.430176 23562 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:22:04.454690 23562 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:22:04.471202 23562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:04.473364 23562 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Loaded TSK: 0
I20260925 15:22:04.475157 23562 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:22:04.708679 23554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:04.708999 23554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:04.709154 23554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:04.716176 23554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:04.716262 23554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:04.716439 23554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:22:04.725533 23554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:43423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=41341
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:04.726693 23554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:04.727974 23554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:04.740947 23581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:04.741144 23580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:04.741906 23583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:04.742141 23554 server_base.cc:1034] running on GCE node
I20260925 15:22:04.742713 23554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:04.743597 23554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:04.744799 23554 hybrid_clock.cc:648] HybridClock initialized: now 1790349724744758 us; error 37 us; skew 500 ppm
I20260925 15:22:04.745128 23554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:04 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:04.748065 23554 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:22:04.749619 23554 webserver.cc:460] Webserver started at http://127.8.226.65:41341/ using document root <none> and password file <none>
I20260925 15:22:04.750270 23554 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:04.750398 23554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:04.755483 23554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:22:04.758139 23590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:04.759454 23554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:04.759619 23554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "4418058d4a094d56976003b0966ec38a"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "67f6916f762e1693409170452aba0fd6"
server_key_iv: "6855f78ea53ae7ae182651c75ad4d392"
server_key_version: "encryptionkey@0"
I20260925 15:22:04.760540 23554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:04.787328 23554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:04.791546 23554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:04.791806 23554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:04.792695 23554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:04.794366 23597 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 15:22:04.799376 23597 ts_tablet_manager.cc:1852] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 15:22:04.800305 23597 ts_tablet_manager.cc:1905] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:04.807147 23597 ts_tablet_manager.cc:1918] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:04.807389 23597 ts_tablet_manager.cc:1939] T 171804c6c23a4b58b11a4e2ff758f46b P 4418058d4a094d56976003b0966ec38a: Deleting consensus metadata
I20260925 15:22:04.807736 23554 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 15:22:04.807855 23554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260925 15:22:04.807981 23554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:04.808090 23554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:04.839002 23554 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43423
I20260925 15:22:04.839020 23703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43423 every 8 connection(s)
I20260925 15:22:04.840241 23554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:04.849620 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23554
I20260925 15:22:04.850365 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:41901
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:34559
--webserver_port=41847
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:22:04 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:04.863780 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:04.843045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:45109)
0925 15:22:04.843822 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:04.843864 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:04.845654 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:04.846820 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:04.846827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:04.847310 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:04.847773 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:04.847788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:04.849310 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:04.849316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:04.851468 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:04.851487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:04.852512 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:04.852522 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:04.852831 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:04.853565 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:04.853596 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:04.856448 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:04.859748 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:04.859758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:04.859772 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:04.860162 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:04.860583 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:04.860588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:04.860591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:04.860751 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:04.861461 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:04.861468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:04.861778 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 15:22:04.862058 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 15:22:04.862543 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":328,"threads_started":1}
I20260925 15:22:04.863791 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:04.842912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:45109 (local address 127.8.226.126:34559)
0925 15:22:04.843438 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:04.843449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:04.843981 (+ 532us) server_negotiation.cc:407] Connection header received
0925 15:22:04.846099 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:04.846112 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:04.846423 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:04.846716 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:04.847939 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:04.849129 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:04.851609 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:04.852294 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:04.852427 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:04.856726 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:04.856733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:04.856745 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:04.856777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:04.859609 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:04.860299 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:04.860312 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:04.860317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:04.860465 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:04.860856 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:04.860862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:04.860865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:04.861336 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:04.861522 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:04.862262 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 15:22:04.862618 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260925 15:22:04.866521 23704 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34559
I20260925 15:22:04.866843 23704 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:04.867637 23704 heartbeater.cc:511] Master 127.8.226.126:34559 requested a full tablet report, sending...
I20260925 15:22:04.869830 23517 ts_manager.cc:194] Registered new tserver with Master: 4418058d4a094d56976003b0966ec38a (127.8.226.65:43423)
I20260925 15:22:04.871241 23517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:45109
W20260925 15:22:05.163925 23708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:05.164213 23708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:05.164383 23708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:05.171267 23708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:05.171393 23708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:05.171546 23708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:22:05.180538 23708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=41847
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:05.181695 23708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:05.182915 23708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:05.195156 23714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:05.195118 23713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:05.195209 23716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:05.195909 23708 server_base.cc:1034] running on GCE node
I20260925 15:22:05.196470 23708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:05.197364 23708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:05.198565 23708 hybrid_clock.cc:648] HybridClock initialized: now 1790349725198531 us; error 43 us; skew 500 ppm
I20260925 15:22:05.198863 23708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:05 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:05.202003 23708 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:22:05.203380 23708 webserver.cc:460] Webserver started at http://127.8.226.66:41847/ using document root <none> and password file <none>
I20260925 15:22:05.204038 23708 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:05.204197 23708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:05.209517 23708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:22:05.211971 23723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:05.213263 23708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:05.213419 23708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "573a89add19848d7bc80c02ef23bcdfe"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "a3a3c6f7ed7a917f94a5770dc8b3dc8e"
server_key_iv: "2de3cea861f9f4c65972bcc2bd7ec7ce"
server_key_version: "encryptionkey@0"
I20260925 15:22:05.214308 23708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:05.228588 23708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:05.232636 23708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:05.232901 23708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:05.233711 23708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:05.234838 23708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:05.234926 23708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:05.234997 23708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:05.235054 23708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:05.265612 23708 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:41901
I20260925 15:22:05.265699 23835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:41901 every 8 connection(s)
I20260925 15:22:05.266958 23708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:05.273665 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23708
I20260925 15:22:05.274386 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:40363
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:34559
--webserver_port=36549
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:22:05 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:05.287146 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.269338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:52921 (local address 127.8.226.126:34559)
0925 15:22:05.269515 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.269520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.270577 (+ 1057us) server_negotiation.cc:407] Connection header received
0925 15:22:05.272161 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.272166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.272238 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.272319 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:05.274072 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.274843 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.277327 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.277712 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.277806 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.281637 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:05.281642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:05.281645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:05.281675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:05.284252 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:05.285023 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:05.285032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:05.285035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:05.285159 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:05.285589 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:05.285593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:05.285596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:05.285838 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:05.285979 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.286593 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.286816 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:22:05.288009 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.269784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:52921)
0925 15:22:05.270423 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.270458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.271940 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.272498 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.272505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.272948 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:05.273907 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.273933 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.274999 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.275008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.277193 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.277210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.277920 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.277929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.278262 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.278953 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:05.278986 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:05.281454 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:05.284395 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:05.284403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:05.284417 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:05.284886 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:05.285283 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:05.285288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:05.285292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:05.285480 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:05.285956 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:05.285963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:05.286263 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.286494 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.286954 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":250,"threads_started":1}
I20260925 15:22:05.289444 23836 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34559
I20260925 15:22:05.289726 23836 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:05.290361 23836 heartbeater.cc:511] Master 127.8.226.126:34559 requested a full tablet report, sending...
I20260925 15:22:05.291684 23517 ts_manager.cc:194] Registered new tserver with Master: 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901)
I20260925 15:22:05.292515 23517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:52921
W20260925 15:22:05.585410 23839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:05.585682 23839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:05.585822 23839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:05.592291 23839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:05.592427 23839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:05.592595 23839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:22:05.601027 23839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:40363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=36549
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:05.602062 23839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:05.603211 23839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:05.615281 23844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:05.615325 23845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:05.615363 23847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:05.616151 23839 server_base.cc:1034] running on GCE node
I20260925 15:22:05.616699 23839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:05.617589 23839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:05.618791 23839 hybrid_clock.cc:648] HybridClock initialized: now 1790349725618749 us; error 53 us; skew 500 ppm
I20260925 15:22:05.619059 23839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:05 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:05.621847 23839 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:22:05.623081 23839 webserver.cc:460] Webserver started at http://127.8.226.67:36549/ using document root <none> and password file <none>
I20260925 15:22:05.623685 23839 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:05.623823 23839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:05.628638 23839 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:22:05.631130 23854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:05.632313 23839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:05.632471 23839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "df3b2f05be1a4b8baa671f8f5fff2701"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "42564ba0a58de91e683fb6e82d8c5dab"
server_key_iv: "c94ad7e361e211f42c0670fdac1631d1"
server_key_version: "encryptionkey@0"
I20260925 15:22:05.633415 23839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:05.656210 23839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:05.659636 23839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:05.659883 23839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:05.660780 23839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:05.661983 23839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:05.662114 23839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:05.662249 23839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:05.662329 23839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:05.693389 23839 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40363
I20260925 15:22:05.693487 23966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40363 every 8 connection(s)
I20260925 15:22:05.694734 23839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:05.696931 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 23839
Sep 25 15:22:05 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:05.712391 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.697882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.8.226.126:34559)
0925 15:22:05.698337 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.698344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.698674 (+ 330us) server_negotiation.cc:407] Connection header received
0925 15:22:05.698896 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.698903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.698995 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.699113 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:05.700652 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.701808 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.703189 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.703617 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.703749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.706345 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:05.706350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:05.706353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:05.706379 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:05.709633 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:05.710242 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:05.710251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:05.710255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:05.710391 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:05.710796 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:05.710800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:05.710804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:05.711175 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:05.711462 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.711735 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.711976 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":334,"thread_start_us":248,"threads_started":1}
I20260925 15:22:05.715940 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.697178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:38363 (local address 127.8.226.126:34559)
0925 15:22:05.697406 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.697412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.699525 (+ 2113us) server_negotiation.cc:407] Connection header received
0925 15:22:05.701547 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.701550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.701598 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.701697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:05.703088 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.703781 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.705346 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.705778 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.705897 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.710053 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:05.710058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:05.710063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:05.710094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:05.713194 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:05.714003 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:05.714008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:05.714012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:05.714122 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:05.714590 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:05.714593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:05.714595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:05.714894 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:05.715026 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.715616 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.715747 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:22:05.716751 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.697778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:38363)
0925 15:22:05.699399 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.699433 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.701277 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.701942 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.701954 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.702435 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:05.702938 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.702953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.703935 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.703939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.705213 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.705225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.705972 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.705980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.706234 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.706857 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:05.706875 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:05.709883 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:05.713360 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:05.713369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:05.713381 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:05.713875 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:05.714221 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:05.714225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:05.714228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:05.714480 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:05.715014 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:05.715021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:05.715293 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.715526 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.715818 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"thread_start_us":1102,"threads_started":1}
I20260925 15:22:05.717866 23967 heartbeater.cc:347] Connected to a master server at 127.8.226.126:34559
I20260925 15:22:05.718175 23967 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:05.718789 23967 heartbeater.cc:511] Master 127.8.226.126:34559 requested a full tablet report, sending...
I20260925 15:22:05.720029 23517 ts_manager.cc:194] Registered new tserver with Master: df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
I20260925 15:22:05.720811 23517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:38363
I20260925 15:22:05.726917 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:05.732641 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.727713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.8.226.126:34559)
0925 15:22:05.727852 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.727856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.728306 (+ 450us) server_negotiation.cc:407] Connection header received
0925 15:22:05.728471 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.728474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.728589 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.728694 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:05.729092 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.729757 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.731033 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.731410 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.731492 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.731880 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:05.732263 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:05.732369 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.732435 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.732487 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:22:05.733408 23517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:05.740399 23517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:05.767100 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.751020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.8.226.65:43423)
0925 15:22:05.757878 (+ 6858us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.757891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.758745 (+ 854us) server_negotiation.cc:407] Connection header received
0925 15:22:05.758843 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.758851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.759118 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.759320 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:05.760241 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.761633 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.764437 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.765931 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.766054 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.766224 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.766594 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.766772 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6636,"thread_start_us":6525,"threads_started":1}
I20260925 15:22:05.767124 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.750991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49736)
0925 15:22:05.758362 (+ 7371us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.758395 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.758547 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.759571 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.759578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.759608 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:05.760000 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.760010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.761798 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.761803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.764263 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.764295 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.766133 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.766140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.766328 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.766387 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.766537 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.766743 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7137,"thread_start_us":8816,"threads_started":1}
I20260925 15:22:05.768870 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.760188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40363 (local address 127.0.0.1:36854)
0925 15:22:05.761629 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.761646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.761756 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.762382 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.762387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.762419 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:05.762957 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.762967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.764467 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.764471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.766526 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.766538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.768108 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.768114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.768267 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.768299 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.768450 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.768570 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"thread_start_us":1200,"threads_started":1}
I20260925 15:22:05.769690 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.760479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.8.226.67:40363)
0925 15:22:05.761094 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.761102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.761876 (+ 774us) server_negotiation.cc:407] Connection header received
0925 15:22:05.761974 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.761984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.762187 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.762382 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:05.763128 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.764366 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.766723 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.767894 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.768012 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.768208 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.769189 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.769365 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":307,"threads_started":1}
I20260925 15:22:05.769748 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.751142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41901 (local address 127.0.0.1:56758)
0925 15:22:05.760125 (+ 8983us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.760140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.760250 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.762266 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.762270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.762294 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:05.762793 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.762805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.764684 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.764690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.766844 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.766858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.769023 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.769031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.769193 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.769224 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.769314 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.769428 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8820,"thread_start_us":6850,"threads_started":1}
I20260925 15:22:05.769551 23639 tablet_service.cc:1467] Processing CreateTablet for tablet f73b60593237481f9b7c9c4e60bdbfd8 (DEFAULT_TABLE table=test-table [id=76755840cb9e470b863c7891fa7cc61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:05.770363 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.751648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.8.226.66:41901)
0925 15:22:05.759642 (+ 7994us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.759652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.760342 (+ 690us) server_negotiation.cc:407] Connection header received
0925 15:22:05.761646 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.761655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.761876 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.762039 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:05.763028 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.764555 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.767034 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.768513 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.768617 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.768837 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.769876 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.770031 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7835,"thread_start_us":7702,"threads_started":1}
I20260925 15:22:05.770874 23639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73b60593237481f9b7c9c4e60bdbfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:05.773403 23771 tablet_service.cc:1467] Processing CreateTablet for tablet f73b60593237481f9b7c9c4e60bdbfd8 (DEFAULT_TABLE table=test-table [id=76755840cb9e470b863c7891fa7cc61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:05.774590 23771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73b60593237481f9b7c9c4e60bdbfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:05.774292 23902 tablet_service.cc:1467] Processing CreateTablet for tablet f73b60593237481f9b7c9c4e60bdbfd8 (DEFAULT_TABLE table=test-table [id=76755840cb9e470b863c7891fa7cc61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:05.775524 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73b60593237481f9b7c9c4e60bdbfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:05.781246 23983 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Bootstrap starting.
I20260925 15:22:05.784662 23983 tablet_bootstrap.cc:654] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:05.785894 23983 log.cc:824] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:05.788436 23985 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Bootstrap starting.
I20260925 15:22:05.788800 23983 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: No bootstrap required, opened a new log
I20260925 15:22:05.789213 23983 ts_tablet_manager.cc:1397] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 15:22:05.791369 23985 tablet_bootstrap.cc:654] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:05.792297 23985 log.cc:824] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:05.794692 23985 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: No bootstrap required, opened a new log
I20260925 15:22:05.794930 23985 ts_tablet_manager.cc:1397] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 15:22:05.798331 23983 raft_consensus.cc:348] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.798610 23983 raft_consensus.cc:374] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:05.798736 23983 raft_consensus.cc:729] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4418058d4a094d56976003b0966ec38a, State: Initialized, Role: FOLLOWER
I20260925 15:22:05.798779 23986 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Bootstrap starting.
I20260925 15:22:05.799278 23983 consensus_queue.cc:260] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.800282 23704 heartbeater.cc:503] Master 127.8.226.126:34559 was elected leader, sending a full tablet report...
I20260925 15:22:05.800885 23983 ts_tablet_manager.cc:1428] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260925 15:22:05.802006 23986 tablet_bootstrap.cc:654] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:05.803160 23986 log.cc:824] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:05.803499 23985 raft_consensus.cc:348] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.803954 23985 raft_consensus.cc:374] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:05.804088 23985 raft_consensus.cc:729] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573a89add19848d7bc80c02ef23bcdfe, State: Initialized, Role: FOLLOWER
I20260925 15:22:05.804667 23985 consensus_queue.cc:260] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.805919 23836 heartbeater.cc:503] Master 127.8.226.126:34559 was elected leader, sending a full tablet report...
I20260925 15:22:05.806354 23986 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: No bootstrap required, opened a new log
I20260925 15:22:05.806716 23986 ts_tablet_manager.cc:1397] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 15:22:05.807503 23985 ts_tablet_manager.cc:1428] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 15:22:05.814514 23986 raft_consensus.cc:348] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.814805 23986 raft_consensus.cc:374] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:05.814908 23986 raft_consensus.cc:729] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b2f05be1a4b8baa671f8f5fff2701, State: Initialized, Role: FOLLOWER
I20260925 15:22:05.815367 23986 consensus_queue.cc:260] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.816180 23967 heartbeater.cc:503] Master 127.8.226.126:34559 was elected leader, sending a full tablet report...
I20260925 15:22:05.816579 23986 ts_tablet_manager.cc:1428] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260925 15:22:05.842253 23705 tablet.cc:2369] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:22:05.946219 23968 tablet.cc:2369] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:05.965803 23988 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:05.966138 23988 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.967772 23988 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
I20260925 15:22:05.974785 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.968205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:42097 (local address 127.8.226.66:41901)
0925 15:22:05.968421 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.968430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.969383 (+ 953us) server_negotiation.cc:407] Connection header received
0925 15:22:05.969596 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.969604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.969678 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.969748 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:05.970515 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.971191 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.973073 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.973815 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.973961 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.974049 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.974401 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.974527 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:22:05.974831 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.968354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:35331 (local address 127.8.226.67:40363)
0925 15:22:05.968584 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:05.968592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:05.969436 (+ 844us) server_negotiation.cc:407] Connection header received
0925 15:22:05.969652 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:05.969656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:05.969751 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:05.969844 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:05.970501 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.971223 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.973045 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.973859 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.974048 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.974119 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:05.974448 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 15:22:05.974570 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:22:05.975001 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.968136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41901 (local address 127.8.226.65:42097)
0925 15:22:05.969127 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.969143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.969435 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.969804 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.969812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.969839 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:05.970356 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.970366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.971331 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.971336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.972887 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.972898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.974048 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.974054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.974232 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.974279 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.974353 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.974492 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":81,"spinlock_wait_cycles":11136,"thread_start_us":489,"threads_started":1}
I20260925 15:22:05.975062 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:05.968208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40363 (local address 127.8.226.65:35331)
0925 15:22:05.969168 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:05.969183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:05.969487 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:05.969880 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:05.969886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:05.969918 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:05.970330 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.970343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.971366 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.971370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.972885 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:05.972897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:05.974104 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:05.974109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:05.974236 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:05.974300 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 15:22:05.974406 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:22:05.974524 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":96,"spinlock_wait_cycles":34688,"thread_start_us":574,"threads_started":1}
I20260925 15:22:05.975957 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73b60593237481f9b7c9c4e60bdbfd8" candidate_uuid: "4418058d4a094d56976003b0966ec38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" is_pre_election: true
I20260925 15:22:05.976140 23791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73b60593237481f9b7c9c4e60bdbfd8" candidate_uuid: "4418058d4a094d56976003b0966ec38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573a89add19848d7bc80c02ef23bcdfe" is_pre_election: true
I20260925 15:22:05.976446 23922 raft_consensus.cc:2454] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4418058d4a094d56976003b0966ec38a in term 0.
I20260925 15:22:05.976615 23791 raft_consensus.cc:2454] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4418058d4a094d56976003b0966ec38a in term 0.
I20260925 15:22:05.977089 23594 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4418058d4a094d56976003b0966ec38a, df3b2f05be1a4b8baa671f8f5fff2701; no voters:
I20260925 15:22:05.977461 23988 raft_consensus.cc:2781] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:05.977605 23988 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:05.977705 23988 raft_consensus.cc:3037] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:05.979743 23988 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.980253 23988 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 1 election: Requested vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
I20260925 15:22:05.980625 23791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73b60593237481f9b7c9c4e60bdbfd8" candidate_uuid: "4418058d4a094d56976003b0966ec38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573a89add19848d7bc80c02ef23bcdfe"
I20260925 15:22:05.980742 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73b60593237481f9b7c9c4e60bdbfd8" candidate_uuid: "4418058d4a094d56976003b0966ec38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b2f05be1a4b8baa671f8f5fff2701"
I20260925 15:22:05.980857 23791 raft_consensus.cc:3037] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:05.980945 23922 raft_consensus.cc:3037] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:05.984022 23791 raft_consensus.cc:2454] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4418058d4a094d56976003b0966ec38a in term 1.
I20260925 15:22:05.984019 23922 raft_consensus.cc:2454] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4418058d4a094d56976003b0966ec38a in term 1.
I20260925 15:22:05.984411 23594 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4418058d4a094d56976003b0966ec38a, df3b2f05be1a4b8baa671f8f5fff2701; no voters:
I20260925 15:22:05.984680 23988 raft_consensus.cc:2781] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:05.985158 23988 raft_consensus.cc:686] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 LEADER]: Becoming Leader. State: Replica: 4418058d4a094d56976003b0966ec38a, State: Running, Role: LEADER
I20260925 15:22:05.985658 23988 consensus_queue.cc:237] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:05.989915 23517 catalog_manager.cc:5495] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4418058d4a094d56976003b0966ec38a (127.8.226.65). New cstate: current_term: 1 leader_uuid: "4418058d4a094d56976003b0966ec38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } health_report { overall_health: UNKNOWN } } }
W20260925 15:22:06.018743 23837 tablet.cc:2369] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:06.057991 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.053395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.8.226.65:43423)
0925 15:22:06.053574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:06.053579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:06.053595 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:22:06.053678 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:06.053681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:06.053762 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:06.053893 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:06.054174 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.054881 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.056194 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.056558 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.056642 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:06.057079 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:06.057534 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:06.057639 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:06.057726 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:22:06.057786 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:22:06.065656 23922 raft_consensus.cc:1261] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Refusing update from remote peer 4418058d4a094d56976003b0966ec38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:06.065656 23791 raft_consensus.cc:1261] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Refusing update from remote peer 4418058d4a094d56976003b0966ec38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:06.066727 23994 consensus_queue.cc:1035] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:06.067081 23988 consensus_queue.cc:1035] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:06.077917 23997 mvcc.cc:205] Tried to move back new op lower bound from 7333272477954834432 to 7333272477640794112. Current Snapshot: MvccSnapshot[applied={T|T < 7333272477954834432}]
I20260925 15:22:06.092337 23516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:06.092654 23516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:06.096163 23516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Sending DeleteTablet for 3 replicas of tablet f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:06.097187 23771 tablet_service.cc:1514] Processing DeleteTablet for tablet f73b60593237481f9b7c9c4e60bdbfd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:06 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:56758
I20260925 15:22:06.097220 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet f73b60593237481f9b7c9c4e60bdbfd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:06 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:36854
I20260925 15:22:06.097242 23639 tablet_service.cc:1514] Processing DeleteTablet for tablet f73b60593237481f9b7c9c4e60bdbfd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:06 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:49736
I20260925 15:22:06.097580 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23554
I20260925 15:22:06.099200 24009 tablet_replica.cc:330] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: stopping tablet replica
I20260925 15:22:06.099617 24009 raft_consensus.cc:2229] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:06.100136 24009 raft_consensus.cc:2258] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:06.103610 24009 ts_tablet_manager.cc:1905] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:06.105408 24007 tablet_replica.cc:330] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: stopping tablet replica
I20260925 15:22:06.105687 24008 tablet_replica.cc:330] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: stopping tablet replica
I20260925 15:22:06.105773 24007 raft_consensus.cc:2229] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:06.106178 24008 raft_consensus.cc:2229] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:22:06.106238 24007 raft_consensus.cc:2258] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:06.108084 24007 ts_tablet_manager.cc:1905] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:06.111737 24009 ts_tablet_manager.cc:1918] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:06.111980 24009 log.cc:1196] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:06.112563 24009 ts_tablet_manager.cc:1939] T f73b60593237481f9b7c9c4e60bdbfd8 P 573a89add19848d7bc80c02ef23bcdfe: Deleting consensus metadata
I20260925 15:22:06.114235 23503 catalog_manager.cc:4841] TS 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): tablet f73b60593237481f9b7c9c4e60bdbfd8 (table test-table [id=76755840cb9e470b863c7891fa7cc61d]) successfully deleted
I20260925 15:22:06.114799 24007 ts_tablet_manager.cc:1918] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:06.114951 24007 log.cc:1196] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:06.115496 24007 ts_tablet_manager.cc:1939] T f73b60593237481f9b7c9c4e60bdbfd8 P df3b2f05be1a4b8baa671f8f5fff2701: Deleting consensus metadata
I20260925 15:22:06.115773 24008 raft_consensus.cc:2258] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:06.117091 23504 catalog_manager.cc:4841] TS df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): tablet f73b60593237481f9b7c9c4e60bdbfd8 (table test-table [id=76755840cb9e470b863c7891fa7cc61d]) successfully deleted
I20260925 15:22:06.117507 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.098575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.8.226.65:43423)
0925 15:22:06.098704 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:06.098710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:06.098728 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:22:06.098886 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:06.098891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:06.098970 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:06.099064 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:06.101138 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.102422 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.105072 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.105744 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.105910 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:06.109593 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:06.109607 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:06.109648 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:06.109699 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:06.113608 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:06.114633 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:06.114641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:06.114645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:06.114821 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:06.115300 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:06.115306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:06.115309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:06.115813 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:06.116189 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:06.116815 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 15:22:06.117084 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:22:06.118173 24008 ts_tablet_manager.cc:1905] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:06.118407 23699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:06.212150 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23554
W20260925 15:22:06.230376 23504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:43423 (error 108)
I20260925 15:22:06.230654 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23708
I20260925 15:22:06.231990 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.231596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49754)
0925 15:22:06.231754 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.231851 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:22:06.232218 23504 catalog_manager.cc:4584] TS 4418058d4a094d56976003b0966ec38a (127.8.226.65:43423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f73b60593237481f9b7c9c4e60bdbfd8: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
I20260925 15:22:06.243918 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.231484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.8.226.66:41901)
0925 15:22:06.231683 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:06.231688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:06.231752 (+ 64us) server_negotiation.cc:407] Connection header received
0925 15:22:06.231948 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:06.231953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:06.232024 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:06.232130 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:06.233619 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.234372 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.235633 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.236038 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.236166 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:06.238666 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:06.238684 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:06.238697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:06.238737 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:06.241272 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:06.241937 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:06.241951 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:06.241953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:06.242070 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:06.242456 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:06.242461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:06.242465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:06.242820 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:06.243091 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:06.243392 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:22:06.243597 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 15:22:06.244504 23831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:06.285517 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.285117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49758)
0925 15:22:06.285293 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.285381 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:22:06.286401 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.286088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49768)
0925 15:22:06.286230 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.286292 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:22:06.331875 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23708
I20260925 15:22:06.347029 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.346469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49772)
0925 15:22:06.346688 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.346809 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:22:06.348269 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.347846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49784)
0925 15:22:06.348023 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.348119 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:22:06.348873 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23839
I20260925 15:22:06.368794 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.349563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.8.226.67:40363)
0925 15:22:06.349738 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:06.349743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:06.349760 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:22:06.349956 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:06.349962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:06.350036 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:06.350134 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:06.351484 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.352196 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.353562 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:06.353950 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:06.354068 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:06.363334 (+ 9266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:06.363350 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:06.363366 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:06.363395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:06.366142 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:06.366833 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:06.366841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:06.366843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:06.366960 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:06.367317 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:06.367324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:06.367327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:06.367673 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:06.367952 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:06.368234 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:22:06.368464 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:22:06.369395 23962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:06.429051 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.428462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49794)
0925 15:22:06.428716 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.428820 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 15:22:06.430027 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:06.429666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43423 (local address 127.0.0.1:49810)
0925 15:22:06.429851 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:06.429923 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 15:22:06.454993 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23839
I20260925 15:22:06.470887 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 23484
I20260925 15:22:06.471932 23547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:06.582238 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 23484
I20260925 15:22:06.602691 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34559
--webserver_interface=127.8.226.126
--webserver_port=45119
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:34559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:06.916288 24015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:06.916581 24015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:06.916736 24015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:06.923372 24015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:06.923507 24015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:06.923609 24015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:06.923704 24015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:06.923789 24015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:06.932755 24015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:34559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=45119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:06.934042 24015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:06.935276 24015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:06.946197 24021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:06.946246 24023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:06.946270 24020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:06.947329 24015 server_base.cc:1034] running on GCE node
I20260925 15:22:06.948035 24015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:06.949435 24015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:06.950646 24015 hybrid_clock.cc:648] HybridClock initialized: now 1790349726950605 us; error 50 us; skew 500 ppm
I20260925 15:22:06.950958 24015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:06 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:06.953841 24015 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:22:06.955118 24015 webserver.cc:460] Webserver started at http://127.8.226.126:45119/ using document root <none> and password file <none>
I20260925 15:22:06.955776 24015 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:06.955915 24015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:06.960820 24015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:22:06.963320 24030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:06.964493 24015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:06.964653 24015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "f9bfa36fdb864f889eb268b754e731cd"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "34e66099619369576c1f238693baa4c4"
server_key_iv: "c6a7058c927db5141f264c5aee232459"
server_key_version: "encryptionkey@0"
I20260925 15:22:06.965626 24015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:06.982088 24015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:06.985857 24015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:06.986097 24015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:07.003314 24015 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:34559
I20260925 15:22:07.003343 24081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:34559 every 8 connection(s)
I20260925 15:22:07.004726 24015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:07.011174 24082 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:22:07.012920 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24015
I20260925 15:22:07.013850 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:43423
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:34559
--webserver_port=41341
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:07.014986 24082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap starting.
I20260925 15:22:07.035801 24082 log.cc:824] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:07.056905 24082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:07.057437 24082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd: Bootstrap complete.
I20260925 15:22:07.065965 24082 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:07.066777 24082 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bfa36fdb864f889eb268b754e731cd, State: Initialized, Role: FOLLOWER
I20260925 15:22:07.067391 24082 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:07.067605 24082 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:07.067708 24082 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:07.067879 24082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:22:07.071704 24082 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:07.072163 24082 leader_election.cc:302] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9bfa36fdb864f889eb268b754e731cd; no voters:
I20260925 15:22:07.072786 24082 leader_election.cc:288] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:22:07.072916 24086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:22:07.075016 24082 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:07.079586 24086 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [term 3 LEADER]: Becoming Leader. State: Replica: f9bfa36fdb864f889eb268b754e731cd, State: Running, Role: LEADER
I20260925 15:22:07.080181 24086 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } }
I20260925 15:22:07.082863 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f9bfa36fdb864f889eb268b754e731cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } } }
I20260925 15:22:07.083213 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:07.083289 24088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bfa36fdb864f889eb268b754e731cd. Latest consensus state: current_term: 3 leader_uuid: "f9bfa36fdb864f889eb268b754e731cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bfa36fdb864f889eb268b754e731cd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34559 } } }
I20260925 15:22:07.083703 24088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bfa36fdb864f889eb268b754e731cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:07.092996 24099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:07.100811 24099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:07.102236 24099 catalog_manager.cc:1259] Loaded cluster ID: bfe910b5bdc34efdb32647d81f9e9cae
I20260925 15:22:07.102322 24099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:22:07.104210 24099 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01aa4259ad google::LogMessage::Fail() at ??:0
@ 0x7f01aa42888a google::LogMessage::SendToLog() at ??:0
@ 0x7f01aa4254de google::LogMessage::Flush() at ??:0
@ 0x7f01aa426a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f01bfe58512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f01bfe55976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f01bfedd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f01bfedd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f01ad071345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f01ad07491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f01ad074799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f01ad04feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01b03296db start_thread at ??:0
@ 0x7f01a92bf61f clone at ??:0
W20260925 15:22:07.395046 24084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:07.395306 24084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:07.395398 24084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:07.402287 24084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:07.402410 24084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:07.402529 24084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:22:07.411804 24084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:43423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=41341
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:07.413035 24084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:07.414261 24084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:07.427285 24123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:07.427341 24124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:07.427255 24126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:07.428054 24084 server_base.cc:1034] running on GCE node
I20260925 15:22:07.428561 24084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:07.429435 24084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:07.430615 24084 hybrid_clock.cc:648] HybridClock initialized: now 1790349727430567 us; error 46 us; skew 500 ppm
I20260925 15:22:07.430907 24084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:07 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:07.433745 24084 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:22:07.435000 24084 webserver.cc:460] Webserver started at http://127.8.226.65:41341/ using document root <none> and password file <none>
I20260925 15:22:07.435601 24084 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:07.435691 24084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:07.440551 24084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:07.443238 24133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:07.444434 24084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:22:07.444594 24084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "4418058d4a094d56976003b0966ec38a"
format_stamp: "Formatted at 2026-09-25 15:22:01 on dist-test-slave-8vw7"
server_key: "67f6916f762e1693409170452aba0fd6"
server_key_iv: "6855f78ea53ae7ae182651c75ad4d392"
server_key_version: "encryptionkey@0"
I20260925 15:22:07.445575 24084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:07.464536 24084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:07.467975 24084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:07.468251 24084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:07.469162 24084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:07.470785 24140 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 15:22:07.475795 24084 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:22:07.475905 24084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 15:22:07.476022 24084 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:22:07.478969 24084 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:22:07.479066 24084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260925 15:22:07.479358 24140 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Bootstrap starting.
I20260925 15:22:07.487030 24140 log.cc:824] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:07.501354 24140 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:07.502192 24140 tablet_bootstrap.cc:492] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Bootstrap complete.
I20260925 15:22:07.502760 24140 ts_tablet_manager.cc:1397] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20260925 15:22:07.511740 24140 raft_consensus.cc:348] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:07.513265 24140 raft_consensus.cc:729] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4418058d4a094d56976003b0966ec38a, State: Initialized, Role: FOLLOWER
I20260925 15:22:07.513955 24140 consensus_queue.cc:260] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:07.515499 24140 ts_tablet_manager.cc:1428] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:22:07.522091 24084 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43423
I20260925 15:22:07.522133 24248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43423 every 8 connection(s)
I20260925 15:22:07.523303 24084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:07.527521 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:07.525875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:39685)
0925 15:22:07.526554 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:07.526786 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":279,"threads_started":1}
W20260925 15:22:07.528164 24249 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:34559 (0 consecutive failures): Network error: Failed to ping master at 127.8.226.126:34559: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
I20260925 15:22:07.528453 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24084
I20260925 15:22:07.528895 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:07.528620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:33355)
0925 15:22:07.528748 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:07.528810 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:22:07.529065 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:41901
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:34559
--webserver_port=41847
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:07.529747 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:07.529367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:35433)
0925 15:22:07.529509 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:07.529590 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:22:07.529973 24249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:22:07.843895 24252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:07.844143 24252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:07.844305 24252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:07.851387 24252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:07.851477 24252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:07.851657 24252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:22:07.860587 24252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=41847
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:07.861727 24252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:07.862913 24252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:07.875228 24260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:07.875262 24258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:07.875289 24257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:07.875984 24252 server_base.cc:1034] running on GCE node
I20260925 15:22:07.876500 24252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:07.877355 24252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:07.878552 24252 hybrid_clock.cc:648] HybridClock initialized: now 1790349727878510 us; error 38 us; skew 500 ppm
I20260925 15:22:07.878839 24252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:07 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:07.881824 24252 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:22:07.883102 24252 webserver.cc:460] Webserver started at http://127.8.226.66:41847/ using document root <none> and password file <none>
I20260925 15:22:07.883837 24252 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:07.883948 24252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:07.888840 24252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:07.891253 24267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:07.892427 24252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:07.892587 24252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "573a89add19848d7bc80c02ef23bcdfe"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "a3a3c6f7ed7a917f94a5770dc8b3dc8e"
server_key_iv: "2de3cea861f9f4c65972bcc2bd7ec7ce"
server_key_version: "encryptionkey@0"
I20260925 15:22:07.893556 24252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:07.918154 24252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:07.922430 24252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:07.922708 24252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:07.923686 24252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:07.925118 24252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:07.925256 24252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:07.925385 24252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:07.925470 24252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:07.958887 24252 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:41901
I20260925 15:22:07.958967 24379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:41901 every 8 connection(s)
I20260925 15:22:07.960306 24252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:07.964893 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:07.963125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:52833)
0925 15:22:07.963813 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:07.964041 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":262,"threads_started":1}
W20260925 15:22:07.965569 24380 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:34559 (0 consecutive failures): Network error: Failed to ping master at 127.8.226.126:34559: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
I20260925 15:22:07.966380 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:07.966042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:58021)
0925 15:22:07.966204 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:07.966282 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:22:07.967132 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:07.966831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:53129)
0925 15:22:07.966961 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:07.967034 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:22:07.967345 24380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:22:07.969779 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24252
I20260925 15:22:07.970355 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:40363
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:34559
--webserver_port=36549
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:08.296226 24383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:08.296552 24383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:08.296715 24383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:08.303496 24383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:08.303622 24383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:08.303795 24383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:22:08.312636 24383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:40363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=36549
--tserver_master_addrs=127.8.226.126:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:08.313761 24383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:08.314980 24383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:08.327728 24391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:08.327768 24388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:08.327791 24389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:08.328720 24383 server_base.cc:1034] running on GCE node
I20260925 15:22:08.329381 24383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:08.330279 24383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:08.331482 24383 hybrid_clock.cc:648] HybridClock initialized: now 1790349728331449 us; error 29 us; skew 500 ppm
I20260925 15:22:08.331768 24383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:08 dist-test-slave-8vw7 krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349728, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:08.334775 24383 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:22:08.336198 24383 webserver.cc:460] Webserver started at http://127.8.226.67:36549/ using document root <none> and password file <none>
I20260925 15:22:08.336859 24383 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:08.337016 24383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:08.342111 24383 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:22:08.344636 24398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:08.345945 24383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:22:08.346068 24383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "df3b2f05be1a4b8baa671f8f5fff2701"
format_stamp: "Formatted at 2026-09-25 15:22:02 on dist-test-slave-8vw7"
server_key: "42564ba0a58de91e683fb6e82d8c5dab"
server_key_iv: "c94ad7e361e211f42c0670fdac1631d1"
server_key_version: "encryptionkey@0"
I20260925 15:22:08.346959 24383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:08.370963 24383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:08.375670 24383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:08.375895 24383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:08.376820 24383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:08.378168 24383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:08.378263 24383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:08.378343 24383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:08.378397 24383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:08.409598 24383 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40363
I20260925 15:22:08.409680 24510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40363 every 8 connection(s)
I20260925 15:22:08.411026 24383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:08.413023 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24383
I20260925 15:22:08.416276 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.414470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:56399)
0925 15:22:08.415204 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.415443 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":280,"threads_started":1}
W20260925 15:22:08.416878 24511 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:34559 (0 consecutive failures): Network error: Failed to ping master at 127.8.226.126:34559: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
I20260925 15:22:08.417659 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.417349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:37911)
0925 15:22:08.417494 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.417563 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:22:08.418473 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.418113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:46537)
0925 15:22:08.418260 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.418354 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 15:22:08.418684 24511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:22:08.531061 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.530418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:39127)
0925 15:22:08.530809 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.530888 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":181,"threads_started":1}
I20260925 15:22:08.896438 24516 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:08.896739 24516 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:08.898365 24516 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
Sep 25 15:22:08 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
Sep 25 15:22:08 dist-test-slave-8vw7 krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:22:08.919569 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.898679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40363 (local address 127.8.226.65:58345)
0925 15:22:08.899005 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.899044 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:08.900023 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:08.903660 (+ 3637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:08.903671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:08.904496 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:08.905321 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:08.905340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.906819 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.906822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:08.908068 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:08.908079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.909068 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.909076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:08.909342 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:08.910149 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:08.910165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:08.912937 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:08.916073 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:08.916082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:08.916096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:08.916616 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:08.917065 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:08.917069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:08.917073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:08.917264 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:08.918047 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:08.918055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:08.918386 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 15:22:08.918645 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 15:22:08.919129 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":225,"spinlock_wait_cycles":13568}
I20260925 15:22:08.919602 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.898908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:35533 (local address 127.8.226.66:41901)
0925 15:22:08.900864 (+ 1956us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:08.900884 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:08.901729 (+ 845us) server_negotiation.cc:407] Connection header received
0925 15:22:08.903150 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:08.903156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:08.903564 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:08.903873 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:08.905492 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.906618 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.908262 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.908874 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.909095 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:08.913048 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:08.913059 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:08.913077 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:08.913117 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:08.916300 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:08.917021 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:08.917030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:08.917037 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:08.917249 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:08.917627 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:08.917630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:08.917632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:08.918080 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:08.918252 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:08.918855 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 15:22:08.919196 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1738,"thread_start_us":1601,"threads_started":1}
I20260925 15:22:08.919649 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.898988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:58345 (local address 127.8.226.67:40363)
0925 15:22:08.901380 (+ 2392us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:08.901392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:08.901434 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:22:08.902860 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:08.902868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:08.903223 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:08.903499 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:08.905500 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.906640 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.908239 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.908862 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.909061 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:08.913168 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:08.913184 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:08.913195 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:08.913226 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:08.915939 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:08.916743 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:08.916752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:08.916759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:08.916944 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:08.917373 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:08.917378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:08.917381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:08.917918 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:08.918048 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:08.918874 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 15:22:08.919256 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200,"thread_start_us":2061,"threads_started":1}
I20260925 15:22:08.919993 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.898740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41901 (local address 127.8.226.65:35533)
0925 15:22:08.901608 (+ 2868us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.901621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:08.901776 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:08.903851 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:08.903856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:08.904496 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:08.905321 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:08.905340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.906805 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.906809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:08.908070 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:08.908081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:08.909106 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:08.909112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:08.909346 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:08.910073 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:08.910108 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:08.912752 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:08.916412 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:08.916415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:08.916417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:08.916841 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:08.917374 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:08.917379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:08.917382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:08.917528 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:08.918211 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:08.918214 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:08.918384 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 15:22:08.918646 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:22:08.919300 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2765,"mutex_wait_us":282,"spinlock_wait_cycles":12544,"thread_start_us":2585,"threads_started":1}
W20260925 15:22:08.923163 24136 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
W20260925 15:22:08.923851 24137 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:08.923971 24137 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4418058d4a094d56976003b0966ec38a; no voters: 573a89add19848d7bc80c02ef23bcdfe, df3b2f05be1a4b8baa671f8f5fff2701
I20260925 15:22:08.924283 24516 raft_consensus.cc:2726] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:22:08.968477 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:08.967746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:50991)
0925 15:22:08.968145 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:08.968274 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":191,"threads_started":1}
I20260925 15:22:09.420104 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:09.419193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:51803)
0925 15:22:09.419610 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:09.419748 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":208,"threads_started":1}
I20260925 15:22:09.532428 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:09.531833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:60371)
0925 15:22:09.532214 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:09.532287 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":188,"threads_started":1}
I20260925 15:22:09.969956 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:09.969360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:42329)
0925 15:22:09.969746 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:09.969817 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":207,"threads_started":1}
I20260925 15:22:10.421576 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:10.420869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:33085)
0925 15:22:10.421324 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:10.421412 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":262,"threads_started":1}
I20260925 15:22:10.533818 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:10.533187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:51249)
0925 15:22:10.533594 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:10.533664 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":198,"threads_started":1}
I20260925 15:22:10.789582 24526 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:10.789778 24526 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:10.790297 24526 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
W20260925 15:22:10.791272 24137 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
W20260925 15:22:10.791426 24136 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:10.791558 24136 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4418058d4a094d56976003b0966ec38a; no voters: 573a89add19848d7bc80c02ef23bcdfe, df3b2f05be1a4b8baa671f8f5fff2701
I20260925 15:22:10.791769 24526 raft_consensus.cc:2726] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:22:10.971287 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:10.970684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:53371)
0925 15:22:10.971075 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:10.971145 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":214,"threads_started":1}
I20260925 15:22:11.422911 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:11.422329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:35769)
0925 15:22:11.422690 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:11.422770 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":193,"threads_started":1}
I20260925 15:22:11.535233 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:11.534610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:56551)
0925 15:22:11.534999 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:11.535071 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":216,"threads_started":1}
I20260925 15:22:11.972764 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:11.972148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:44795)
0925 15:22:11.972526 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:11.972595 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":198,"threads_started":1}
I20260925 15:22:12.424446 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:12.423728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:46529)
0925 15:22:12.424200 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:12.424272 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260925 15:22:12.536733 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:12.536069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:47759)
0925 15:22:12.536490 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:12.536562 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":196,"threads_started":1}
I20260925 15:22:12.974110 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:12.973532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:46925)
0925 15:22:12.973895 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:12.973965 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":190,"threads_started":1}
I20260925 15:22:13.058503 24534 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:13.058733 24534 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:13.059382 24534 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
W20260925 15:22:13.060250 24136 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
W20260925 15:22:13.060390 24137 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:13.060506 24137 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4418058d4a094d56976003b0966ec38a; no voters: 573a89add19848d7bc80c02ef23bcdfe, df3b2f05be1a4b8baa671f8f5fff2701
I20260925 15:22:13.060704 24534 raft_consensus.cc:2726] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:22:13.425923 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:13.425220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:55849)
0925 15:22:13.425647 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:13.425729 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":256,"threads_started":1}
I20260925 15:22:13.538286 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:13.537524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:49431)
0925 15:22:13.538000 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:13.538103 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260925 15:22:13.975508 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:13.974864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:58847)
0925 15:22:13.975281 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:13.975354 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260925 15:22:14.427596 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:14.426768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:36531)
0925 15:22:14.427269 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:14.427344 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":311,"threads_started":1}
I20260925 15:22:14.539700 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:14.539047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:53005)
0925 15:22:14.539430 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:14.539514 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260925 15:22:14.977152 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:14.976365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:42683)
0925 15:22:14.976832 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:14.976904 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":208,"threads_started":1}
I20260925 15:22:15.429103 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:15.428371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:54799)
0925 15:22:15.428801 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:15.428871 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":201,"threads_started":1}
I20260925 15:22:15.541298 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:15.540610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:40743)
0925 15:22:15.541066 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:15.541142 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260925 15:22:15.978636 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:15.977948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:59531)
0925 15:22:15.978387 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:15.978463 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260925 15:22:16.430510 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:16.429871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:33731)
0925 15:22:16.430256 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:16.430333 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":196,"threads_started":1}
I20260925 15:22:16.542976 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:16.542087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:39291)
0925 15:22:16.542736 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:16.542813 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":444,"threads_started":1}
I20260925 15:22:16.625972 24546 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:16.626255 24546 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:16.626956 24546 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
W20260925 15:22:16.627923 24136 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
W20260925 15:22:16.628098 24137 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:16.628226 24137 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4418058d4a094d56976003b0966ec38a; no voters: 573a89add19848d7bc80c02ef23bcdfe, df3b2f05be1a4b8baa671f8f5fff2701
I20260925 15:22:16.628444 24546 raft_consensus.cc:2726] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:22:16.980254 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:16.979496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:51485)
0925 15:22:16.979995 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:16.980102 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260925 15:22:17.432217 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:17.431468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:47091)
0925 15:22:17.431983 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:17.432067 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":277,"threads_started":1}
I20260925 15:22:17.544474 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:17.543834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:39453)
0925 15:22:17.544247 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:17.544320 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":219,"threads_started":1}
I20260925 15:22:17.981782 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:17.981155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:39871)
0925 15:22:17.981531 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:17.981606 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":199,"threads_started":1}
I20260925 15:22:18.433779 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:18.433054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:36365)
0925 15:22:18.433532 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:18.433605 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":273,"threads_started":1}
I20260925 15:22:18.546029 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:18.545355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:46655)
0925 15:22:18.545779 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:18.545860 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":229,"threads_started":1}
I20260925 15:22:18.983361 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:18.982620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:53307)
0925 15:22:18.983084 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:18.983186 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":221,"threads_started":1}
I20260925 15:22:19.435190 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:19.434570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:50073)
0925 15:22:19.434976 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:19.435048 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":203,"threads_started":1}
I20260925 15:22:19.547513 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:19.546800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:45579)
0925 15:22:19.547298 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:19.547370 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
I20260925 15:22:19.984943 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:19.984198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:56783)
0925 15:22:19.984685 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:19.984778 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":219,"threads_started":1}
I20260925 15:22:20.437146 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:20.436246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:59519)
0925 15:22:20.436819 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:20.436915 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":300,"threads_started":1}
I20260925 15:22:20.549080 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:20.548314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:57189)
0925 15:22:20.548769 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:20.548855 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260925 15:22:20.986492 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:20.985831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:59293)
0925 15:22:20.986249 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:20.986332 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":212,"threads_started":1}
I20260925 15:22:21.438809 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:21.438093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:54157)
0925 15:22:21.438542 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:21.438655 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260925 15:22:21.550625 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:21.549907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:43415)
0925 15:22:21.550303 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:21.550461 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":199,"threads_started":1}
I20260925 15:22:21.988204 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:21.987366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:56449)
0925 15:22:21.987903 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:21.987979 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":280,"threads_started":1}
I20260925 15:22:22.363613 24563 raft_consensus.cc:482] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:22.363819 24563 raft_consensus.cc:504] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4418058d4a094d56976003b0966ec38a" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43423 } } peers { permanent_uuid: "573a89add19848d7bc80c02ef23bcdfe" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41901 } } peers { permanent_uuid: "df3b2f05be1a4b8baa671f8f5fff2701" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40363 } }
I20260925 15:22:22.364460 24563 leader_election.cc:288] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901), df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363)
W20260925 15:22:22.365445 24137 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b2f05be1a4b8baa671f8f5fff2701 (127.8.226.67:40363): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
W20260925 15:22:22.365617 24136 leader_election.cc:341] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 573a89add19848d7bc80c02ef23bcdfe (127.8.226.66:41901): Not found: Tablet not found: f73b60593237481f9b7c9c4e60bdbfd8
I20260925 15:22:22.365787 24136 leader_election.cc:302] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4418058d4a094d56976003b0966ec38a; no voters: 573a89add19848d7bc80c02ef23bcdfe, df3b2f05be1a4b8baa671f8f5fff2701
I20260925 15:22:22.366073 24563 raft_consensus.cc:2726] T f73b60593237481f9b7c9c4e60bdbfd8 P 4418058d4a094d56976003b0966ec38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:22:22.440306 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:22.439591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:33897)
0925 15:22:22.440032 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:22.440122 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260925 15:22:22.552407 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:22.551636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.65:58485)
0925 15:22:22.552093 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:22.552200 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":202,"threads_started":1}
I20260925 15:22:22.990056 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:22.989291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.66:45287)
0925 15:22:22.989793 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:22.989894 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":281,"threads_started":1}
I20260925 15:22:23.418134 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24084
I20260925 15:22:23.433632 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:23.419238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.8.226.65:43423)
0925 15:22:23.419795 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:23.419803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:23.419829 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:22:23.419894 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:23.419900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:23.420123 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:23.420350 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:23.422664 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:23.423600 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:23.424862 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:23.425591 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:23.425737 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:23.428278 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:23.428294 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:23.428313 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:23.428343 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:23.430939 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:23.431615 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:23.431622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:23.431624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:23.431768 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:23.432136 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:23.432142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:23.432146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:23.432472 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:23.432790 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:23.433048 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:22:23.433304 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260925 15:22:23.434944 24244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:23.442116 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:23.441373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:34559 (local address 127.8.226.67:50001)
0925 15:22:23.441886 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:23.441978 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":285,"threads_started":1}
I20260925 15:22:23.523098 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24084
I20260925 15:22:23.540414 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24252
I20260925 15:22:23.555994 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:23.541122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.8.226.66:41901)
0925 15:22:23.541535 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:23.541540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:23.541556 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:22:23.541623 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:23.541626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:23.541686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:23.541773 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:23.543257 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:23.544518 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:23.546221 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:23.546816 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:23.546921 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:23.550421 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:23.550429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:23.550433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:23.550471 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:23.553444 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:23.554109 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:23.554115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:23.554119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:23.554258 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:23.554624 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:23.554628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:23.554631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:23.554871 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:23.555139 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:23.555409 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 15:22:23.555623 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":209,"threads_started":1}
I20260925 15:22:23.556638 24375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:23.641381 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24252
I20260925 15:22:23.660231 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24383
I20260925 15:22:23.673681 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:23.661139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.8.226.67:40363)
0925 15:22:23.661737 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:23.661742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:23.661760 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:22:23.661817 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:23.661821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:23.661918 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:23.662011 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:23.663414 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:23.664184 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:23.665487 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:23.665888 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:23.665994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:23.668486 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:23.668491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:23.668494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:23.668523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:23.671260 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:23.671933 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:23.671936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:23.671938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:23.672070 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:23.672397 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:23.672400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:23.672402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:23.672722 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:23.672919 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:23.673222 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:22:23.673394 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":319,"threads_started":1}
I20260925 15:22:23.674391 24506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:23.755990 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24383
2026-09-25T15:22:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24041 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:22:23 dist-test-slave-8vw7 krb5kdc[24576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:22:23 dist-test-slave-8vw7 krb5kdc[24576](info): set up 2 sockets
Sep 25 15:22:23 dist-test-slave-8vw7 krb5kdc[24576](info): commencing operation
krb5kdc: starting...
W20260925 15:22:24.853924 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:22:24 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:22:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:22:25.080554 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44275
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:44275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:25.402238 24589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:25.402557 24589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:25.402724 24589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:25.409724 24589 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:22:25.409852 24589 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:22:25.409940 24589 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:22:25.410032 24589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:25.410133 24589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:25.410251 24589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:25.410334 24589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:25.410414 24589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:25.419869 24589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:44275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:25.421109 24589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:25.422420 24589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:25.434003 24594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:25.434059 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:25.434082 24597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:25.435190 24589 server_base.cc:1034] running on GCE node
I20260925 15:22:25.435930 24589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:25.437314 24589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:25.438522 24589 hybrid_clock.cc:648] HybridClock initialized: now 1790349745438467 us; error 28 us; skew 500 ppm
I20260925 15:22:25.438856 24589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:25 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349745, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:25.446539 24589 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:22:25.447868 24589 webserver.cc:460] Webserver started at http://127.8.226.126:38951/ using document root <none> and password file <none>
I20260925 15:22:25.448465 24589 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:25.448542 24589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:25.448787 24589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:25.451664 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "49de814a44a9452385f7881cbfea51db"
format_stamp: "Formatted at 2026-09-25 15:22:25 on dist-test-slave-8vw7"
server_key: "fa038b25dd870fd93fc7a90cc5bd0ba4"
server_key_iv: "2794ec83e05af9c559a30985defed4c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:25.452369 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "49de814a44a9452385f7881cbfea51db"
format_stamp: "Formatted at 2026-09-25 15:22:25 on dist-test-slave-8vw7"
server_key: "fa038b25dd870fd93fc7a90cc5bd0ba4"
server_key_iv: "2794ec83e05af9c559a30985defed4c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:25.457324 24589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:22:25.460510 24604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:25.461786 24589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:25.461911 24589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "49de814a44a9452385f7881cbfea51db"
format_stamp: "Formatted at 2026-09-25 15:22:25 on dist-test-slave-8vw7"
server_key: "fa038b25dd870fd93fc7a90cc5bd0ba4"
server_key_iv: "2794ec83e05af9c559a30985defed4c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:25.462065 24589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:25.480612 24589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:25.484254 24589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:25.484460 24589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:25.502472 24589 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:44275
I20260925 15:22:25.502494 24655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:44275 every 8 connection(s)
I20260925 15:22:25.503765 24589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:25.507228 24656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:25.511659 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24589
I20260925 15:22:25.511883 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:22:25.512544 9097 external_mini_cluster.cc:1428] Setting key d029a10ff7ad25f315ed8326ef97218e
I20260925 15:22:25.517392 24656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap starting.
I20260925 15:22:25.520921 24656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:25.522235 24656 log.cc:824] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:25.524684 24656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: No bootstrap required, opened a new log
Sep 25 15:22:25 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:25.532469 24656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:25.532822 24656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:25.532924 24656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49de814a44a9452385f7881cbfea51db, State: Initialized, Role: FOLLOWER
I20260925 15:22:25.533578 24656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:25.533812 24656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:25.533999 24656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:25.534186 24656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:25.535557 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:25.514457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.8.226.126:44275)
0925 15:22:25.515349 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:25.515362 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:25.515403 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:22:25.517076 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:25.517096 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:25.517461 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:25.517798 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:25.519751 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:25.521048 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:25.523034 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:25.523693 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:25.523881 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:25.528068 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:25.528075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:25.528092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:25.528124 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:25.531375 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:25.532080 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:25.532096 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:25.532101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:25.532273 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:25.532673 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:25.532681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:25.532684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:25.533380 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:25.533679 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:25.534110 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 15:22:25.534578 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":424,"threads_started":1}
I20260925 15:22:25.536651 24656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:25.537249 24656 leader_election.cc:302] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49de814a44a9452385f7881cbfea51db; no voters:
I20260925 15:22:25.537858 24656 leader_election.cc:288] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:22:25.537967 24661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:25.539139 24661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 LEADER]: Becoming Leader. State: Replica: 49de814a44a9452385f7881cbfea51db, State: Running, Role: LEADER
I20260925 15:22:25.539728 24656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:25.539603 24661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:25.542883 24662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49de814a44a9452385f7881cbfea51db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } } }
I20260925 15:22:25.543182 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49de814a44a9452385f7881cbfea51db. Latest consensus state: current_term: 1 leader_uuid: "49de814a44a9452385f7881cbfea51db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } } }
I20260925 15:22:25.543467 24662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:25.543530 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:25.545243 24669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:25.550436 24669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:25.560499 24669 catalog_manager.cc:1347] Generated new cluster ID: 0beff253563b471da978748d5240075d
I20260925 15:22:25.560596 24669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:25.591791 24669 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:22:25.593592 24669 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:25.604800 24669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Generated new TSK 0
I20260925 15:22:25.605829 24669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:22:25.664191 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44275
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:25.984544 24682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:25.984805 24682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:25.984894 24682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:25.991775 24682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:25.991869 24682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:25.991976 24682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:22:26.001211 24682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:26.002345 24682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:26.003542 24682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:26.016244 24690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:26.016279 24688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:26.016813 24687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:26.017181 24682 server_base.cc:1034] running on GCE node
I20260925 15:22:26.017731 24682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:26.018591 24682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:26.019826 24682 hybrid_clock.cc:648] HybridClock initialized: now 1790349746019786 us; error 35 us; skew 500 ppm
I20260925 15:22:26.020117 24682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:26 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349746, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:26.023124 24682 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:22:26.024443 24682 webserver.cc:460] Webserver started at http://127.8.226.65:43265/ using document root <none> and password file <none>
I20260925 15:22:26.025216 24682 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:26.025315 24682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:26.025576 24682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:26.028259 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "777078cf323148fe95dc73599d77bd6e"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "1c41aa650c7e7210913f7adcac0eb217"
server_key_iv: "d518cc3378d08c4942500fc7018c32a2"
server_key_version: "encryptionkey@0"
I20260925 15:22:26.028930 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "777078cf323148fe95dc73599d77bd6e"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "1c41aa650c7e7210913f7adcac0eb217"
server_key_iv: "d518cc3378d08c4942500fc7018c32a2"
server_key_version: "encryptionkey@0"
I20260925 15:22:26.033843 24682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:22:26.037154 24697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:26.038360 24682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:22:26.038530 24682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "777078cf323148fe95dc73599d77bd6e"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "1c41aa650c7e7210913f7adcac0eb217"
server_key_iv: "d518cc3378d08c4942500fc7018c32a2"
server_key_version: "encryptionkey@0"
I20260925 15:22:26.038686 24682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:26.052021 24682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:26.055809 24682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:26.056063 24682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:26.056934 24682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:26.058182 24682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:26.058276 24682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:26.058408 24682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:26.058470 24682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:26.090730 24682 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:42451
I20260925 15:22:26.090770 24809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:42451 every 8 connection(s)
I20260925 15:22:26.091985 24682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:26.096200 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24682
I20260925 15:22:26.096439 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:22:26.097349 9097 external_mini_cluster.cc:1428] Setting key 366b804f2654583abb1550f68624983d
Sep 25 15:22:26 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349746, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:26.113440 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:26.094588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:34861 (local address 127.8.226.126:44275)
0925 15:22:26.095043 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:26.095049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:26.096123 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 15:22:26.098610 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:26.098615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:26.098685 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:26.098801 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:26.100602 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.101369 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.103833 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.104364 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.104500 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:26.107920 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:26.107924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:26.107927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:26.107957 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:26.110528 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:26.111223 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:26.111230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:26.111237 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:26.111350 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:26.111767 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:26.111771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:26.111774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:26.112025 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:26.112192 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:26.112822 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 15:22:26.113062 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":214,"threads_started":1}
I20260925 15:22:26.114337 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:26.095171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:34861)
0925 15:22:26.095927 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:26.095978 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:26.098324 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:26.098981 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:26.098988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:26.099688 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:26.100436 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:26.100462 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.101512 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.101520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:26.103601 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:26.103617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.104577 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.104585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:26.104824 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:26.105608 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:26.105628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:26.107692 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:26.110671 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:26.110679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:26.110694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:26.111094 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:26.111459 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:26.111463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:26.111467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:26.111642 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:26.112151 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:26.112160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:26.112476 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 15:22:26.112748 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 15:22:26.113271 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":251,"threads_started":1}
I20260925 15:22:26.115851 24810 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44275
I20260925 15:22:26.116185 24810 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:26.117063 24810 heartbeater.cc:511] Master 127.8.226.126:44275 requested a full tablet report, sending...
I20260925 15:22:26.119545 24621 ts_manager.cc:194] Registered new tserver with Master: 777078cf323148fe95dc73599d77bd6e (127.8.226.65:42451)
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
I20260925 15:22:26.121806 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:34861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:22:26.150394 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44275
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:26.466645 24816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:26.466914 24816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:26.467005 24816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:26.474097 24816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:26.474195 24816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:26.474306 24816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:22:26.483613 24816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:26.484825 24816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:26.486078 24816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:26.499066 24821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:26.499090 24822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:26.499665 24816 server_base.cc:1034] running on GCE node
W20260925 15:22:26.499773 24824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:26.500437 24816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:26.501276 24816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:26.502480 24816 hybrid_clock.cc:648] HybridClock initialized: now 1790349746502425 us; error 49 us; skew 500 ppm
I20260925 15:22:26.502792 24816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:26 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349746, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:26.506376 24816 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:22:26.510075 24816 webserver.cc:460] Webserver started at http://127.8.226.66:36421/ using document root <none> and password file <none>
I20260925 15:22:26.510691 24816 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:26.510842 24816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:26.511137 24816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:26.513875 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "e134e97e296f4cf2be9954dcf07613d2"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "ce6c710d113a624443f42d4c83f233ee"
server_key_iv: "4d86f66f7354ba368924eb8789aa0361"
server_key_version: "encryptionkey@0"
I20260925 15:22:26.514581 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "e134e97e296f4cf2be9954dcf07613d2"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "ce6c710d113a624443f42d4c83f233ee"
server_key_iv: "4d86f66f7354ba368924eb8789aa0361"
server_key_version: "encryptionkey@0"
I20260925 15:22:26.519289 24816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:22:26.522492 24831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:26.523701 24816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:22:26.523872 24816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "e134e97e296f4cf2be9954dcf07613d2"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "ce6c710d113a624443f42d4c83f233ee"
server_key_iv: "4d86f66f7354ba368924eb8789aa0361"
server_key_version: "encryptionkey@0"
I20260925 15:22:26.524051 24816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:26.541622 24816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:26.545822 24816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:26.546103 24816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:26.546986 24816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:26.548166 24816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:26.548261 24816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:26.548404 24816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:26.548475 24816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:26.580578 24816 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:38837
I20260925 15:22:26.580669 24943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:38837 every 8 connection(s)
I20260925 15:22:26.581815 24816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:26.591059 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24816
I20260925 15:22:26.591253 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:22:26.591688 9097 external_mini_cluster.cc:1428] Setting key e4465b273b10486e69de0766a9d819c4
Sep 25 15:22:26 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349746, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:26.600989 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:26.584169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:46247 (local address 127.8.226.126:44275)
0925 15:22:26.584355 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:26.584360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:26.585563 (+ 1203us) server_negotiation.cc:407] Connection header received
0925 15:22:26.587332 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:26.587337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:26.587434 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:26.587544 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:26.589006 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.589730 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.591307 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.591752 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.591890 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:26.595559 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:26.595563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:26.595567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:26.595595 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:26.598113 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:26.598801 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:26.598808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:26.598814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:26.598944 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:26.599350 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:26.599356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:26.599360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:26.599610 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:26.599774 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:26.600403 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 15:22:26.600615 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:22:26.601917 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:26.584643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:46247)
0925 15:22:26.585408 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:26.585448 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:26.587062 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:26.587779 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:26.587786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:26.588267 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:26.588798 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:26.588814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.589874 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.589881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:26.591087 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:26.591098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:26.591977 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:26.591984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:26.592268 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:26.593025 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:26.593059 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:26.595334 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:26.598255 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:26.598264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:26.598278 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:26.598661 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:26.599052 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:26.599058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:26.599061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:26.599240 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:26.599735 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:26.599742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:26.600056 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 15:22:26.600327 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 15:22:26.600801 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260925 15:22:26.603348 24944 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44275
I20260925 15:22:26.603631 24944 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:26.604317 24944 heartbeater.cc:511] Master 127.8.226.126:44275 requested a full tablet report, sending...
I20260925 15:22:26.605818 24621 ts_manager.cc:194] Registered new tserver with Master: e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
I20260925 15:22:26.607049 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:46247
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:22:26.641422 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44275
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:26.955704 24949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:26.956028 24949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:26.956197 24949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:26.963291 24949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:26.963425 24949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:26.963598 24949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:22:26.972846 24949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:26.974048 24949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:26.975281 24949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:26.990129 24955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:26.990092 24954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:26.990801 24957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:26.991477 24949 server_base.cc:1034] running on GCE node
I20260925 15:22:26.992245 24949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:26.993470 24949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:26.994719 24949 hybrid_clock.cc:648] HybridClock initialized: now 1790349746994612 us; error 80 us; skew 500 ppm
I20260925 15:22:26.995122 24949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:26 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349746, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:26.999015 24949 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:22:27.000452 24949 webserver.cc:460] Webserver started at http://127.8.226.67:34151/ using document root <none> and password file <none>
I20260925 15:22:27.001155 24949 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:27.001304 24949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:27.001610 24949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:27.004299 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "2b3361739e3142c59746d0bc39d8d1ec"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "34687d96c6f5a04fe16e855f091e97f2"
server_key_iv: "1de19a7df32fa2685420bda949f76460"
server_key_version: "encryptionkey@0"
I20260925 15:22:27.005132 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "2b3361739e3142c59746d0bc39d8d1ec"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "34687d96c6f5a04fe16e855f091e97f2"
server_key_iv: "1de19a7df32fa2685420bda949f76460"
server_key_version: "encryptionkey@0"
I20260925 15:22:27.010131 24949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:22:27.013576 24964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:27.014921 24949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:22:27.015090 24949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "2b3361739e3142c59746d0bc39d8d1ec"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "34687d96c6f5a04fe16e855f091e97f2"
server_key_iv: "1de19a7df32fa2685420bda949f76460"
server_key_version: "encryptionkey@0"
I20260925 15:22:27.015286 24949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:27.030395 24949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:27.034452 24949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:27.034715 24949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:27.035651 24949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:27.041733 24949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:27.041837 24949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:27.041970 24949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:27.042047 24949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:27.079066 24949 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:38693
I20260925 15:22:27.079111 25076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:38693 every 8 connection(s)
I20260925 15:22:27.080325 24949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:27.082724 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 24949
I20260925 15:22:27.082919 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:22:27.083346 9097 external_mini_cluster.cc:1428] Setting key 1e4257bcecdf8a65cb44af752334bdd8
Sep 25 15:22:27 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349746, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:27.099504 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.082845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:56271 (local address 127.8.226.126:44275)
0925 15:22:27.083035 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.083041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.084316 (+ 1275us) server_negotiation.cc:407] Connection header received
0925 15:22:27.086142 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.086148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.086219 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.086316 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:27.087676 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.088426 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.090025 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.090601 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.090742 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.094129 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:27.094134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:27.094137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:27.094166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:27.096590 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.097352 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.097357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.097359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.097482 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.097875 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.097883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.097885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.098204 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:27.098367 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.098952 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.099154 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:22:27.100303 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.083390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:56271)
0925 15:22:27.084143 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:27.084204 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:27.085872 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:27.086513 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:27.086522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:27.086991 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:27.087528 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.087544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.088552 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.088556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.089874 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.089886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.090802 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.090807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.090985 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.091639 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:27.091663 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:27.093937 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:27.096725 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:27.096734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:27.096748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:27.097224 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:27.097587 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:27.097592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:27.097596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:27.097762 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:27.098326 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:27.098333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:27.098635 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 15:22:27.098864 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 15:22:27.099321 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":311,"threads_started":1}
I20260925 15:22:27.101509 25077 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44275
I20260925 15:22:27.101742 25077 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:27.102383 25077 heartbeater.cc:511] Master 127.8.226.126:44275 requested a full tablet report, sending...
I20260925 15:22:27.103672 24621 ts_manager.cc:194] Registered new tserver with Master: 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693)
I20260925 15:22:27.104396 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:56271
I20260925 15:22:27.110437 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:27.124799 24810 heartbeater.cc:503] Master 127.8.226.126:44275 was elected leader, sending a full tablet report...
I20260925 15:22:27.126839 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.114796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.8.226.126:44275)
0925 15:22:27.114981 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.114989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.115344 (+ 355us) server_negotiation.cc:407] Connection header received
0925 15:22:27.115532 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.115538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.115601 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.115705 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:27.117094 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.117770 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.119249 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.119671 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.119772 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.121890 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:27.121894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:27.121896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:27.121936 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:27.124130 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.124765 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.124772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.124775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.124909 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.125348 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.125353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.125356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.125689 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:27.125860 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.126278 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.126500 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:22:27.130316 24621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:27.136869 24621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:27.166761 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.148064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38693 (local address 127.0.0.1:55372)
0925 15:22:27.156503 (+ 8439us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:27.156525 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:27.156662 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:27.158389 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:27.158400 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:27.158441 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:27.158874 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.158889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.161464 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.161478 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.163841 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.163850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.165635 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.165641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.165852 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.165910 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:22:27.166049 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:22:27.166218 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8281,"mutex_wait_us":4,"thread_start_us":8135,"threads_started":1}
I20260925 15:22:27.167239 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.148675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.8.226.67:38693)
0925 15:22:27.157581 (+ 8906us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.157591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.157628 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:22:27.157740 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.157751 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.158041 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.158319 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:27.159166 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.161279 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.163981 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.165412 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.165554 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.165790 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.166692 (+ 902us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.166951 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8689,"thread_start_us":8494,"threads_started":1}
I20260925 15:22:27.169754 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.148569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.8.226.66:38837)
0925 15:22:27.158960 (+ 10391us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.158967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.158992 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:22:27.160300 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.160311 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.160587 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.160789 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:27.162541 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.163985 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.166643 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.168373 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.168502 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.168690 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.169227 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.169389 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10234,"thread_start_us":10131,"threads_started":1}
I20260925 15:22:27.170292 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.159026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:42734)
0925 15:22:27.162422 (+ 3396us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:27.162439 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:27.162549 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:27.163084 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:27.163089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:27.163119 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:27.163501 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.163513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.165375 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.165380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.167579 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.167593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.169400 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.169406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.169556 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.169610 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:22:27.169705 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:22:27.169965 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3234,"thread_start_us":3060,"threads_started":1}
I20260925 15:22:27.170573 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.147926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:38837 (local address 127.0.0.1:44144)
0925 15:22:27.154491 (+ 6565us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:27.154553 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:27.154752 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:27.161114 (+ 6362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:27.161121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:27.161154 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:27.162364 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.162375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.164330 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.164336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.166480 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.166493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.168816 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.168822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.169011 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.169047 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:22:27.169988 (+ 941us) client_negotiation.cc:240] Negotiation successful
0925 15:22:27.170127 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6238,"mutex_wait_us":682,"thread_start_us":6090,"threads_started":1}
I20260925 15:22:27.171171 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.157072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.8.226.65:42451)
0925 15:22:27.162137 (+ 5065us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.162146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.162571 (+ 425us) server_negotiation.cc:407] Connection header received
0925 15:22:27.162678 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.162684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.162896 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.163072 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:27.163684 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.165222 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.167765 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.169208 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.169337 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.169538 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.169760 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.169913 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4837,"thread_start_us":4660,"threads_started":1}
I20260925 15:22:27.171108 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 17229396db66442aae55a86bf5811e74 (DEFAULT_TABLE table=test-table [id=8620f5cc0628473ea23d3915d801acc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:27.172403 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17229396db66442aae55a86bf5811e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:27.173511 24879 tablet_service.cc:1467] Processing CreateTablet for tablet 17229396db66442aae55a86bf5811e74 (DEFAULT_TABLE table=test-table [id=8620f5cc0628473ea23d3915d801acc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:27.174527 24879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17229396db66442aae55a86bf5811e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:27.179487 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 17229396db66442aae55a86bf5811e74 (DEFAULT_TABLE table=test-table [id=8620f5cc0628473ea23d3915d801acc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:27.180665 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17229396db66442aae55a86bf5811e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:27.184737 25095 tablet_bootstrap.cc:492] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Bootstrap starting.
I20260925 15:22:27.186574 25096 tablet_bootstrap.cc:492] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Bootstrap starting.
I20260925 15:22:27.189383 25095 tablet_bootstrap.cc:654] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:27.190002 25096 tablet_bootstrap.cc:654] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:27.190640 25095 log.cc:824] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:27.191125 25096 log.cc:824] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:27.192572 25099 tablet_bootstrap.cc:492] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Bootstrap starting.
I20260925 15:22:27.193725 25095 tablet_bootstrap.cc:492] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: No bootstrap required, opened a new log
I20260925 15:22:27.193951 25096 tablet_bootstrap.cc:492] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: No bootstrap required, opened a new log
I20260925 15:22:27.193980 25095 ts_tablet_manager.cc:1397] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 15:22:27.194209 25096 ts_tablet_manager.cc:1397] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260925 15:22:27.195438 25099 tablet_bootstrap.cc:654] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:27.196538 25099 log.cc:824] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:27.198906 25099 tablet_bootstrap.cc:492] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: No bootstrap required, opened a new log
I20260925 15:22:27.199151 25099 ts_tablet_manager.cc:1397] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 15:22:27.204551 25095 raft_consensus.cc:348] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.204550 25096 raft_consensus.cc:348] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.204983 25095 raft_consensus.cc:374] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:27.204974 25096 raft_consensus.cc:374] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:27.205070 25095 raft_consensus.cc:729] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e134e97e296f4cf2be9954dcf07613d2, State: Initialized, Role: FOLLOWER
I20260925 15:22:27.205077 25096 raft_consensus.cc:729] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3361739e3142c59746d0bc39d8d1ec, State: Initialized, Role: FOLLOWER
I20260925 15:22:27.205653 25096 consensus_queue.cc:260] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.205657 25095 consensus_queue.cc:260] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.205706 25099 raft_consensus.cc:348] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.206014 25099 raft_consensus.cc:374] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:27.206110 25099 raft_consensus.cc:729] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777078cf323148fe95dc73599d77bd6e, State: Initialized, Role: FOLLOWER
I20260925 15:22:27.206795 25099 consensus_queue.cc:260] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.207337 24944 heartbeater.cc:503] Master 127.8.226.126:44275 was elected leader, sending a full tablet report...
I20260925 15:22:27.207523 25096 ts_tablet_manager.cc:1428] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:22:27.207784 25095 ts_tablet_manager.cc:1428] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260925 15:22:27.207962 25077 heartbeater.cc:503] Master 127.8.226.126:44275 was elected leader, sending a full tablet report...
I20260925 15:22:27.210628 25099 ts_tablet_manager.cc:1428] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
W20260925 15:22:27.331987 25078 tablet.cc:2369] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:22:27.333542 24945 tablet.cc:2369] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:22:27.344429 24811 tablet.cc:2369] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:27.362406 25103 raft_consensus.cc:482] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:27.362715 25103 raft_consensus.cc:504] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.364364 25103 leader_election.cc:288] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837), 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693)
I20260925 15:22:27.371196 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.364747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:33261 (local address 127.8.226.66:38837)
0925 15:22:27.364995 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.365003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.365907 (+ 904us) server_negotiation.cc:407] Connection header received
0925 15:22:27.366070 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.366075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.366152 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.366258 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:27.366846 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.367552 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.369475 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.370261 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.370446 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.370523 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.370828 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.370949 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:22:27.371390 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.364798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38693 (local address 127.8.226.65:45143)
0925 15:22:27.365938 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:27.365956 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:27.366108 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:27.366445 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:27.366449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:27.366477 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:27.366833 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.366841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.367854 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.367860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.369329 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.369340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.370475 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.370481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.370624 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.370694 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 15:22:27.370795 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:22:27.370915 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":40,"thread_start_us":574,"threads_started":1}
I20260925 15:22:27.371206 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.364927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:45143 (local address 127.8.226.67:38693)
0925 15:22:27.365161 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.365170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.366080 (+ 910us) server_negotiation.cc:407] Connection header received
0925 15:22:27.366251 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.366259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.366325 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.366422 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:27.366983 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.367709 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.369470 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.370244 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.370432 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.370509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.370828 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.370944 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 15:22:27.371474 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.364663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:38837 (local address 127.8.226.65:33261)
0925 15:22:27.365790 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:27.365805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:27.365933 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:27.366269 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:27.366274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:27.366295 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:27.366694 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.366707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.367686 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.367692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.369322 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:27.369334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.370516 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.370522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:27.370671 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.370697 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 15:22:27.370803 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:22:27.370916 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":80,"thread_start_us":888,"threads_started":1}
I20260925 15:22:27.372432 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17229396db66442aae55a86bf5811e74" candidate_uuid: "777078cf323148fe95dc73599d77bd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3361739e3142c59746d0bc39d8d1ec" is_pre_election: true
I20260925 15:22:27.372617 24899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17229396db66442aae55a86bf5811e74" candidate_uuid: "777078cf323148fe95dc73599d77bd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e134e97e296f4cf2be9954dcf07613d2" is_pre_election: true
I20260925 15:22:27.372925 25032 raft_consensus.cc:2454] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777078cf323148fe95dc73599d77bd6e in term 0.
I20260925 15:22:27.373152 24899 raft_consensus.cc:2454] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777078cf323148fe95dc73599d77bd6e in term 0.
I20260925 15:22:27.373674 24701 leader_election.cc:302] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3361739e3142c59746d0bc39d8d1ec, 777078cf323148fe95dc73599d77bd6e; no voters:
I20260925 15:22:27.374060 25103 raft_consensus.cc:2781] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:27.374205 25103 raft_consensus.cc:482] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:27.374320 25103 raft_consensus.cc:3037] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:27.376464 25103 raft_consensus.cc:504] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.377146 25103 leader_election.cc:288] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 1 election: Requested vote from peers e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837), 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693)
I20260925 15:22:27.377461 24899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17229396db66442aae55a86bf5811e74" candidate_uuid: "777078cf323148fe95dc73599d77bd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e134e97e296f4cf2be9954dcf07613d2"
I20260925 15:22:27.377648 24899 raft_consensus.cc:3037] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:27.377652 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17229396db66442aae55a86bf5811e74" candidate_uuid: "777078cf323148fe95dc73599d77bd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3361739e3142c59746d0bc39d8d1ec"
I20260925 15:22:27.377828 25032 raft_consensus.cc:3037] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:27.379779 24899 raft_consensus.cc:2454] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777078cf323148fe95dc73599d77bd6e in term 1.
I20260925 15:22:27.379866 25032 raft_consensus.cc:2454] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777078cf323148fe95dc73599d77bd6e in term 1.
I20260925 15:22:27.380196 24699 leader_election.cc:302] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e, e134e97e296f4cf2be9954dcf07613d2; no voters:
I20260925 15:22:27.380505 25103 raft_consensus.cc:2781] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:27.380920 25103 raft_consensus.cc:686] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 1 LEADER]: Becoming Leader. State: Replica: 777078cf323148fe95dc73599d77bd6e, State: Running, Role: LEADER
I20260925 15:22:27.381417 25103 consensus_queue.cc:237] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } }
I20260925 15:22:27.385810 24620 catalog_manager.cc:5495] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 777078cf323148fe95dc73599d77bd6e (127.8.226.65). New cstate: current_term: 1 leader_uuid: "777078cf323148fe95dc73599d77bd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } health_report { overall_health: HEALTHY } } }
I20260925 15:22:27.453253 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.447692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.8.226.65:42451)
0925 15:22:27.447888 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.447895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.447986 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:22:27.448231 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.448235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.448326 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.448485 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:27.448869 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.449692 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.451128 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.451513 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.451643 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.452129 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:27.452632 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:27.452784 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.452860 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.452932 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:22:27.461257 25032 raft_consensus.cc:1261] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Refusing update from remote peer 777078cf323148fe95dc73599d77bd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:27.461258 24899 raft_consensus.cc:1261] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Refusing update from remote peer 777078cf323148fe95dc73599d77bd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:27.462293 25106 consensus_queue.cc:1035] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:27.462608 25103 consensus_queue.cc:1035] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:27.471503 25109 mvcc.cc:205] Tried to move back new op lower bound from 7333272565591638016 to 7333272565277839360. Current Snapshot: MvccSnapshot[applied={T|T < 7333272565591638016}]
I20260925 15:22:27.478951 25111 mvcc.cc:205] Tried to move back new op lower bound from 7333272565591638016 to 7333272565277839360. Current Snapshot: MvccSnapshot[applied={T|T < 7333272565591638016}]
I20260925 15:22:27.485592 24620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:27.485891 24620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:27.489529 24620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Sending DeleteTablet for 3 replicas of tablet 17229396db66442aae55a86bf5811e74
I20260925 15:22:27.490592 24879 tablet_service.cc:1514] Processing DeleteTablet for tablet 17229396db66442aae55a86bf5811e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:27 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:44144
I20260925 15:22:27.490752 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 17229396db66442aae55a86bf5811e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:27 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:55372
I20260925 15:22:27.491103 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24682
I20260925 15:22:27.491236 24745 tablet_service.cc:1514] Processing DeleteTablet for tablet 17229396db66442aae55a86bf5811e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:27 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:42734
I20260925 15:22:27.492722 25118 tablet_replica.cc:330] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: stopping tablet replica
I20260925 15:22:27.493185 25118 raft_consensus.cc:2229] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:27.493256 25119 tablet_replica.cc:330] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: stopping tablet replica
I20260925 15:22:27.493623 25119 raft_consensus.cc:2229] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:27.493675 25118 raft_consensus.cc:2258] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:27.494076 25119 raft_consensus.cc:2258] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:27.495855 25119 ts_tablet_manager.cc:1905] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:27.497439 25118 ts_tablet_manager.cc:1905] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:27.502090 25120 tablet_replica.cc:330] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: stopping tablet replica
I20260925 15:22:27.502537 25120 raft_consensus.cc:2229] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:22:27.502621 25119 ts_tablet_manager.cc:1918] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:27.502754 25119 log.cc:1196] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/17229396db66442aae55a86bf5811e74
I20260925 15:22:27.503185 25119 ts_tablet_manager.cc:1939] T 17229396db66442aae55a86bf5811e74 P 2b3361739e3142c59746d0bc39d8d1ec: Deleting consensus metadata
I20260925 15:22:27.503237 25120 raft_consensus.cc:2258] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 15:22:27 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:22:27.505151 24608 catalog_manager.cc:4841] TS 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): tablet 17229396db66442aae55a86bf5811e74 (table test-table [id=8620f5cc0628473ea23d3915d801acc7]) successfully deleted
I20260925 15:22:27.506096 25120 ts_tablet_manager.cc:1905] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:27.507228 25118 ts_tablet_manager.cc:1918] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:27.507381 25118 log.cc:1196] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/17229396db66442aae55a86bf5811e74
I20260925 15:22:27.507846 25118 ts_tablet_manager.cc:1939] T 17229396db66442aae55a86bf5811e74 P e134e97e296f4cf2be9954dcf07613d2: Deleting consensus metadata
I20260925 15:22:27.509517 24606 catalog_manager.cc:4841] TS e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): tablet 17229396db66442aae55a86bf5811e74 (table test-table [id=8620f5cc0628473ea23d3915d801acc7]) successfully deleted
I20260925 15:22:27.513808 25120 ts_tablet_manager.cc:1918] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:27.513978 25120 log.cc:1196] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/17229396db66442aae55a86bf5811e74
I20260925 15:22:27.514472 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.493523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.8.226.65:42451)
0925 15:22:27.493692 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.493700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.493722 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:22:27.493777 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.493784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.493869 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.493975 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:27.496093 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.497351 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.499408 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.500115 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.500208 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.506701 (+ 6493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:27.506723 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:27.506740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:27.506785 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:27.510646 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.511577 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.511588 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.511592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.511773 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.512311 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.512316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.512320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.512807 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:27.513132 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.513798 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.514045 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:22:27.514503 25120 ts_tablet_manager.cc:1939] T 17229396db66442aae55a86bf5811e74 P 777078cf323148fe95dc73599d77bd6e: Deleting consensus metadata
I20260925 15:22:27.515863 24805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:27.516177 24608 catalog_manager.cc:4841] TS 777078cf323148fe95dc73599d77bd6e (127.8.226.65:42451): tablet 17229396db66442aae55a86bf5811e74 (table test-table [id=8620f5cc0628473ea23d3915d801acc7]) successfully deleted
I20260925 15:22:27.616436 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24682
I20260925 15:22:27.634999 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24816
Sep 25 15:22:27 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:22:27.649493 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.635763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.8.226.66:38837)
0925 15:22:27.635953 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.635957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.635974 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:22:27.636102 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.636105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.636164 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.636244 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:27.637488 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.638312 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.639625 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.639981 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.640070 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.644224 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:27.644237 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:27.644252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:27.644286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:27.646963 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.647583 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.647589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.647591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.647702 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.647994 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.647997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.647999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.648325 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:27.648520 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.648763 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.648900 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:22:27.650089 24939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:27.736994 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24816
I20260925 15:22:27.753496 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24949
Sep 25 15:22:27 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:22:27.767513 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:27.754183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.8.226.67:38693)
0925 15:22:27.754351 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:27.754356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:27.754426 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:22:27.754566 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:27.754570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:27.754631 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:27.754760 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:27.756089 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.756780 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.758079 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:27.758452 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:27.758541 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:27.762752 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:27.762765 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:27.762781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:27.762811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:27.765320 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.765922 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.765928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.765930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.766060 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:27.766397 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:27.766400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:27.766402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:27.766708 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:27.766904 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:27.767114 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:22:27.767273 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:22:27.768122 25072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:27.855556 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24949
I20260925 15:22:27.874606 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 24589
I20260925 15:22:27.875787 24651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:27.986435 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 24589
I20260925 15:22:48.008102 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44275
--webserver_interface=127.8.226.126
--webserver_port=38951
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:44275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:48.325412 25126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:48.325685 25126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:48.325773 25126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:48.332823 25126 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:22:48.332906 25126 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:22:48.332944 25126 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:22:48.333060 25126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:48.333110 25126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:48.333175 25126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:48.333220 25126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:48.333271 25126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:48.342660 25126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:44275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=38951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:48.343814 25126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:48.344987 25126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:48.367106 25132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.367129 25135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.367065 25133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:48.374655 25126 server_base.cc:1034] running on GCE node
I20260925 15:22:48.375483 25126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:48.377029 25126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:48.378252 25126 hybrid_clock.cc:648] HybridClock initialized: now 1790349768378169 us; error 56 us; skew 500 ppm
I20260925 15:22:48.378661 25126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:48 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:48.387125 25126 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:22:48.388810 25126 webserver.cc:460] Webserver started at http://127.8.226.126:38951/ using document root <none> and password file <none>
I20260925 15:22:48.389715 25126 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:48.389813 25126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:48.395602 25126 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:22:48.398386 25142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.399820 25126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:22:48.399995 25126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "49de814a44a9452385f7881cbfea51db"
format_stamp: "Formatted at 2026-09-25 15:22:25 on dist-test-slave-8vw7"
server_key: "fa038b25dd870fd93fc7a90cc5bd0ba4"
server_key_iv: "2794ec83e05af9c559a30985defed4c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:48.401041 25126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:48.435741 25126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:48.439755 25126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:48.440079 25126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:48.458247 25126 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:44275
I20260925 15:22:48.458287 25193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:44275 every 8 connection(s)
I20260925 15:22:48.459720 25126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:48.460881 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25126
I20260925 15:22:48.461898 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:42451
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:44275
--webserver_port=43265
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:48.469754 25194 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:22:48.473434 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap starting.
I20260925 15:22:48.492511 25194 log.cc:824] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:48.504055 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:48.504634 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap complete.
I20260925 15:22:48.514150 25194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:48.514948 25194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49de814a44a9452385f7881cbfea51db, State: Initialized, Role: FOLLOWER
I20260925 15:22:48.515588 25194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:48.515821 25194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:48.515930 25194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:48.516110 25194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:22:48.519783 25194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:48.520268 25194 leader_election.cc:302] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49de814a44a9452385f7881cbfea51db; no voters:
I20260925 15:22:48.520980 25194 leader_election.cc:288] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:22:48.521114 25198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:22:48.522234 25198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 LEADER]: Becoming Leader. State: Replica: 49de814a44a9452385f7881cbfea51db, State: Running, Role: LEADER
I20260925 15:22:48.522756 25198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:22:48.523275 25194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:48.525107 25200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49de814a44a9452385f7881cbfea51db. Latest consensus state: current_term: 2 leader_uuid: "49de814a44a9452385f7881cbfea51db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } } }
I20260925 15:22:48.525496 25200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:48.525977 25199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "49de814a44a9452385f7881cbfea51db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } } }
I20260925 15:22:48.526288 25199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:48.534917 25204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:48.544293 25204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:48.546031 25204 catalog_manager.cc:1259] Loaded cluster ID: 0beff253563b471da978748d5240075d
I20260925 15:22:48.546156 25204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:48.550853 25204 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:48.552863 25204 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Loaded TSK: 0
I20260925 15:22:48.593160 25204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Generated new TSK 1
I20260925 15:22:48.593638 25204 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:22:48.826426 25196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:48.826683 25196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:48.826820 25196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:48.833909 25196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:48.834012 25196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:48.834182 25196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:22:48.843952 25196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:42451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=43265
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:48.845283 25196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:48.846565 25196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:48.859922 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.860194 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.860862 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:48.860844 25196 server_base.cc:1034] running on GCE node
I20260925 15:22:48.861567 25196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:48.862486 25196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:48.863719 25196 hybrid_clock.cc:648] HybridClock initialized: now 1790349768863658 us; error 56 us; skew 500 ppm
I20260925 15:22:48.864022 25196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:48 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:48.867301 25196 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:22:48.868880 25196 webserver.cc:460] Webserver started at http://127.8.226.65:43265/ using document root <none> and password file <none>
I20260925 15:22:48.869592 25196 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:48.869753 25196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:48.875070 25196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:22:48.877921 25231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.879295 25196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:48.879460 25196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "777078cf323148fe95dc73599d77bd6e"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "1c41aa650c7e7210913f7adcac0eb217"
server_key_iv: "d518cc3378d08c4942500fc7018c32a2"
server_key_version: "encryptionkey@0"
I20260925 15:22:48.880375 25196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:48.899390 25196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:48.903263 25196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:48.903514 25196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:48.904557 25196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:48.905860 25196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:48.905972 25196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.906095 25196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:48.906201 25196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.943106 25196 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:42451
I20260925 15:22:48.943177 25343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:42451 every 8 connection(s)
I20260925 15:22:48.944550 25196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:48.947466 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25196
I20260925 15:22:48.948273 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:38837
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:44275
--webserver_port=36421
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:22:48 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:48.971623 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:48.947606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:44555)
0925 15:22:48.948880 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:48.948937 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:48.951653 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:48.953082 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:48.953097 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:48.953885 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:48.954653 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:48.954680 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.956822 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.956826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:48.958106 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:48.958118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.959567 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.959574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:48.959763 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:48.960439 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:48.960465 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:48.962930 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:48.967659 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:48.967665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:48.967675 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:48.968106 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:48.968644 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:48.968648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:48.968650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:48.968826 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:48.969827 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:48.969833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:48.970132 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 15:22:48.970346 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 15:22:48.970700 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"thread_start_us":756,"threads_started":1}
I20260925 15:22:48.972352 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:48.947529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:44555 (local address 127.8.226.126:44275)
0925 15:22:48.948261 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:48.948278 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:48.949096 (+ 818us) server_negotiation.cc:407] Connection header received
0925 15:22:48.952189 (+ 3093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:48.952201 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:48.952526 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:48.952820 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:48.954878 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.956613 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.958248 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.959348 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.959530 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:48.963211 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:48.963225 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:48.963246 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:48.963300 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:48.967508 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:48.968253 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:48.968264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:48.968274 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:48.968519 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:48.968971 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:48.968980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:48.968983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:48.969683 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:48.969906 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:48.970619 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 15:22:48.971074 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":301,"threads_started":1}
I20260925 15:22:48.975137 25344 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44275
I20260925 15:22:48.975431 25344 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:48.976075 25344 heartbeater.cc:511] Master 127.8.226.126:44275 requested a full tablet report, sending...
I20260925 15:22:48.978152 25159 ts_manager.cc:194] Registered new tserver with Master: 777078cf323148fe95dc73599d77bd6e (127.8.226.65:42451)
I20260925 15:22:48.979631 25159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:44555
W20260925 15:22:49.276095 25348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.276412 25348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.276613 25348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.283805 25348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.283941 25348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.284111 25348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:22:49.294328 25348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:38837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=36421
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:49.295646 25348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.297025 25348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.310019 25353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.309896 25356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.309931 25354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.311010 25348 server_base.cc:1034] running on GCE node
I20260925 15:22:49.311599 25348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.313624 25348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.314860 25348 hybrid_clock.cc:648] HybridClock initialized: now 1790349769314782 us; error 50 us; skew 500 ppm
I20260925 15:22:49.315235 25348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:49 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.318195 25348 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:22:49.319530 25348 webserver.cc:460] Webserver started at http://127.8.226.66:36421/ using document root <none> and password file <none>
I20260925 15:22:49.320185 25348 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.320338 25348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:49.325516 25348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:22:49.328033 25363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.329341 25348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:49.329502 25348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "e134e97e296f4cf2be9954dcf07613d2"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "ce6c710d113a624443f42d4c83f233ee"
server_key_iv: "4d86f66f7354ba368924eb8789aa0361"
server_key_version: "encryptionkey@0"
I20260925 15:22:49.330482 25348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.351877 25348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.355722 25348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.355973 25348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.357036 25348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:49.358336 25348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:49.358454 25348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.358579 25348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:49.358667 25348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.392763 25348 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:38837
I20260925 15:22:49.392846 25475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:38837 every 8 connection(s)
I20260925 15:22:49.394181 25348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:49.395077 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25348
I20260925 15:22:49.395823 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:38693
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:44275
--webserver_port=34151
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:22:49 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:49.415740 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.397109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:37103 (local address 127.8.226.126:44275)
0925 15:22:49.397312 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.397318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.398747 (+ 1429us) server_negotiation.cc:407] Connection header received
0925 15:22:49.401514 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.401518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.401579 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.401663 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.403588 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.404314 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.406024 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.406441 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.406594 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.410119 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.410124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.410127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.410158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.412804 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.413515 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.413523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.413532 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.413640 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.414063 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.414068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.414071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.414347 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.414501 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.415148 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.415398 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:22:49.416684 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.397675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:37103)
0925 15:22:49.398548 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.398601 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.401239 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.401927 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.401940 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.402681 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:49.403427 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.403452 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.404473 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.404478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.405880 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.405891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.406681 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.406688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.406916 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.407585 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:49.407609 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:49.409893 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:49.412933 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.412941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.412984 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.413391 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.413758 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.413763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.413767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.413947 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.414470 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:49.414477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:49.414806 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.415055 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.415575 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
I20260925 15:22:49.418169 25476 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44275
I20260925 15:22:49.418466 25476 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.419130 25476 heartbeater.cc:511] Master 127.8.226.126:44275 requested a full tablet report, sending...
I20260925 15:22:49.420575 25159 ts_manager.cc:194] Registered new tserver with Master: e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
I20260925 15:22:49.421825 25159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:37103
W20260925 15:22:49.739358 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.739627 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.739723 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.746701 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.746805 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.746915 25479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:22:49.756249 25479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:38693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=34151
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:22:49.757450 25479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.758642 25479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.771538 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.771588 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.771585 25485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.772123 25479 server_base.cc:1034] running on GCE node
I20260925 15:22:49.772698 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.773653 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.774845 25479 hybrid_clock.cc:648] HybridClock initialized: now 1790349769774805 us; error 32 us; skew 500 ppm
I20260925 15:22:49.775132 25479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:22:49 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.778052 25479 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:22:49.779417 25479 webserver.cc:460] Webserver started at http://127.8.226.67:34151/ using document root <none> and password file <none>
I20260925 15:22:49.780115 25479 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.780221 25479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:49.785408 25479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:22:49.788146 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.789532 25479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:49.789758 25479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "2b3361739e3142c59746d0bc39d8d1ec"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "34687d96c6f5a04fe16e855f091e97f2"
server_key_iv: "1de19a7df32fa2685420bda949f76460"
server_key_version: "encryptionkey@0"
I20260925 15:22:49.790753 25479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.811113 25479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.815106 25479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.815372 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.816298 25479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:49.817539 25479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:49.817621 25479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.817787 25479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:49.817878 25479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.851092 25479 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:38693
I20260925 15:22:49.851174 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:38693 every 8 connection(s)
I20260925 15:22:49.852504 25479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:49.861222 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25479
Sep 25 15:22:49 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:22:49.872640 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.854932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:51579 (local address 127.8.226.126:44275)
0925 15:22:49.855175 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.855181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.856260 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 15:22:49.857995 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.857999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.858063 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.858148 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.859492 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.860173 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.862124 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.862784 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.862916 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.866684 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.866689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.866692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.866720 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.869659 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.870558 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.870561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.870563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.870668 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.871135 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.871138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.871140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.871373 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.871520 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.872188 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.872386 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:22:49.874053 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.855416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:51579)
0925 15:22:49.856102 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.856142 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.857762 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.858345 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.858354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.858833 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:49.859358 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.859374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.860295 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.860298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.861981 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.861995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.863107 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.863116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.863433 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.864135 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:49.864167 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:49.866474 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:49.869796 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.869806 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.869823 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.870447 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.870774 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.870781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.870785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.871030 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.871500 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:49.871508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:49.871824 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.872054 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.872556 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":284,"threads_started":1}
I20260925 15:22:49.875517 25607 heartbeater.cc:347] Connected to a master server at 127.8.226.126:44275
I20260925 15:22:49.875828 25607 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.876497 25607 heartbeater.cc:511] Master 127.8.226.126:44275 requested a full tablet report, sending...
I20260925 15:22:49.877678 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.862233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.8.226.126:44275)
0925 15:22:49.863112 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.863117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.863134 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:22:49.863183 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.863187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.863271 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.863383 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.865050 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.865870 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.867602 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.868331 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.868450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.871589 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.871595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.871599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.871626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.874702 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.875457 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.875462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.875464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.875600 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.875971 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.875979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.875982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.876233 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.876489 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.876792 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.877038 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":621,"threads_started":1}
I20260925 15:22:49.878032 25159 ts_manager.cc:194] Registered new tserver with Master: 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693)
I20260925 15:22:49.879009 25159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:51579
I20260925 15:22:49.880367 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:49.886507 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.881296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.8.226.126:44275)
0925 15:22:49.881468 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.881473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.881928 (+ 455us) server_negotiation.cc:407] Connection header received
0925 15:22:49.882066 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.882069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.882174 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.882285 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:49.882622 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.883356 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.884893 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.885350 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.885440 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.885891 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:49.886076 (+ 185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:22:49.886329 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39636: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 15:22:49.886587 25611 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39636: token expired
W20260925 15:22:49.887153 25080 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.226.126:44275)
I20260925 15:22:49.899343 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.887640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.8.226.126:44275)
0925 15:22:49.887785 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.887788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.887804 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:22:49.887923 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.887926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.887970 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.888042 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.889382 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.890095 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.891423 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.891784 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.891872 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.894678 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.894683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.894686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.894714 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.897190 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.897800 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.897803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.897805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.897909 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.898243 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.898245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.898247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.898540 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.898704 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.898964 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.899111 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:22:49.902456 25158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:49.909888 25158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:49.930629 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.921263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.8.226.65:42451)
0925 15:22:49.922520 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.922532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.922561 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:22:49.922660 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.922667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.922943 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.924095 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:49.924260 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.925696 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.928010 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.929496 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.929685 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.929892 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.930151 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.930309 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":952,"threads_started":1}
I20260925 15:22:49.930675 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.920995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52680)
0925 15:22:49.921752 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.921797 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.921968 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.923216 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.923228 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.923281 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:49.923837 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.923869 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.925848 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.925856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.927833 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.927847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.929733 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.929739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.929912 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.929968 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.930095 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.930289 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"spinlock_wait_cycles":16000,"thread_start_us":697,"threads_started":1}
I20260925 15:22:49.931798 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.921777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.8.226.66:38837)
0925 15:22:49.922870 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.922880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.923397 (+ 517us) server_negotiation.cc:407] Connection header received
0925 15:22:49.923717 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.923726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.923964 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.924198 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:49.925062 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.926588 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.929130 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.930480 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.930611 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.930784 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.931350 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.931504 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":826,"threads_started":1}
I20260925 15:22:49.931824 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.921566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:38837 (local address 127.0.0.1:37504)
0925 15:22:49.923266 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.923286 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.923517 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.924447 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.924451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.924475 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:49.924842 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.924855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.926740 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.926746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.928926 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.928986 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.930931 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.930938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.931141 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.931186 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.931285 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.931410 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"mutex_wait_us":3,"spinlock_wait_cycles":2944,"thread_start_us":1350,"threads_started":1}
I20260925 15:22:49.932175 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.922437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38693 (local address 127.0.0.1:35992)
0925 15:22:49.923523 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.923537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.923629 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.924797 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.924805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.924835 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:49.925294 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.925306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.927233 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.927241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.929257 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.929271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.931246 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.931254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.931411 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.931439 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.931495 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.931616 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":1,"thread_start_us":828,"threads_started":1}
I20260925 15:22:49.933174 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.922971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.8.226.67:38693)
0925 15:22:49.924199 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.924206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.924234 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:22:49.924320 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.924326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.924561 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.925476 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:49.925749 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.927100 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.929410 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.931020 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.932366 (+ 1346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.932618 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.932685 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.932850 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":896,"threads_started":1}
I20260925 15:22:49.934578 25279 tablet_service.cc:1467] Processing CreateTablet for tablet b712c53790ad4dab98e9607b55cc39d6 (DEFAULT_TABLE table=test-table [id=39d7cbfcdfc443f0abe3a901abc34af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:49.935761 25279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b712c53790ad4dab98e9607b55cc39d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:49.936655 25542 tablet_service.cc:1467] Processing CreateTablet for tablet b712c53790ad4dab98e9607b55cc39d6 (DEFAULT_TABLE table=test-table [id=39d7cbfcdfc443f0abe3a901abc34af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:49.936655 25411 tablet_service.cc:1467] Processing CreateTablet for tablet b712c53790ad4dab98e9607b55cc39d6 (DEFAULT_TABLE table=test-table [id=39d7cbfcdfc443f0abe3a901abc34af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:49.937951 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b712c53790ad4dab98e9607b55cc39d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:49.937963 25411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b712c53790ad4dab98e9607b55cc39d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:49.955107 25623 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Bootstrap starting.
I20260925 15:22:49.957672 25625 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Bootstrap starting.
I20260925 15:22:49.958356 25623 tablet_bootstrap.cc:654] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:49.959641 25623 log.cc:824] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:49.961205 25625 tablet_bootstrap.cc:654] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:49.961635 25624 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Bootstrap starting.
I20260925 15:22:49.962391 25625 log.cc:824] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:49.964054 25623 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: No bootstrap required, opened a new log
I20260925 15:22:49.964315 25623 ts_tablet_manager.cc:1397] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 15:22:49.965819 25624 tablet_bootstrap.cc:654] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:49.967024 25624 log.cc:824] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:49.967468 25625 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: No bootstrap required, opened a new log
I20260925 15:22:49.967759 25625 ts_tablet_manager.cc:1397] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 15:22:49.969900 25624 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: No bootstrap required, opened a new log
I20260925 15:22:49.970146 25624 ts_tablet_manager.cc:1397] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 15:22:49.972497 25623 raft_consensus.cc:348] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:49.972837 25623 raft_consensus.cc:374] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:49.972995 25623 raft_consensus.cc:729] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777078cf323148fe95dc73599d77bd6e, State: Initialized, Role: FOLLOWER
I20260925 15:22:49.973598 25623 consensus_queue.cc:260] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:49.975226 25344 heartbeater.cc:503] Master 127.8.226.126:44275 was elected leader, sending a full tablet report...
I20260925 15:22:49.975898 25623 ts_tablet_manager.cc:1428] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260925 15:22:49.977974 25625 raft_consensus.cc:348] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:49.978360 25625 raft_consensus.cc:374] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:49.978457 25625 raft_consensus.cc:729] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e134e97e296f4cf2be9954dcf07613d2, State: Initialized, Role: FOLLOWER
I20260925 15:22:49.979027 25625 consensus_queue.cc:260] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:49.980124 25476 heartbeater.cc:503] Master 127.8.226.126:44275 was elected leader, sending a full tablet report...
I20260925 15:22:49.980342 25624 raft_consensus.cc:348] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:49.980756 25624 raft_consensus.cc:374] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:49.980782 25625 ts_tablet_manager.cc:1428] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 15:22:49.980861 25624 raft_consensus.cc:729] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3361739e3142c59746d0bc39d8d1ec, State: Initialized, Role: FOLLOWER
I20260925 15:22:49.981416 25624 consensus_queue.cc:260] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:49.982431 25607 heartbeater.cc:503] Master 127.8.226.126:44275 was elected leader, sending a full tablet report...
I20260925 15:22:49.983043 25624 ts_tablet_manager.cc:1428] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260925 15:22:50.051545 25630 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:50.051870 25630 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:50.053467 25630 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777078cf323148fe95dc73599d77bd6e (127.8.226.65:42451), 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693)
I20260925 15:22:50.060072 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.053889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:34587 (local address 127.8.226.65:42451)
0925 15:22:50.054201 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:50.054206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:50.054651 (+ 445us) server_negotiation.cc:407] Connection header received
0925 15:22:50.054879 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:50.054885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:50.054967 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:50.055072 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:50.055688 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.056662 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.058381 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.059165 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.059350 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.059424 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:50.059704 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:22:50.059824 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 15:22:50.060171 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.053844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.8.226.66:34587)
0925 15:22:50.054547 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.054569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:50.054702 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:50.055097 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:50.055102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:50.055128 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:50.055504 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:50.055516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.056788 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.056791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:50.058230 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:50.058241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.059372 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.059378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:50.059531 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.059555 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:22:50.059673 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 15:22:50.059795 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":395,"threads_started":1}
I20260925 15:22:50.060575 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.054838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:55031 (local address 127.8.226.67:38693)
0925 15:22:50.055043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:50.055051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:50.055328 (+ 277us) server_negotiation.cc:407] Connection header received
0925 15:22:50.055526 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:50.055552 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:50.055636 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:50.055735 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:50.056307 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.057217 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.058860 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.059618 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.059829 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.059895 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:50.060201 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:22:50.060337 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:22:50.061246 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.054664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38693 (local address 127.8.226.66:55031)
0925 15:22:50.055230 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.055242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:50.055394 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:50.055765 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:50.055771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:50.055798 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:50.056141 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:50.056150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.057352 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.057358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:50.058718 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:50.058729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.059851 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.059857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:50.060013 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.060035 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:22:50.060770 (+ 735us) client_negotiation.cc:240] Negotiation successful
0925 15:22:50.060886 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":386,"threads_started":1}
I20260925 15:22:50.061419 25299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b712c53790ad4dab98e9607b55cc39d6" candidate_uuid: "e134e97e296f4cf2be9954dcf07613d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777078cf323148fe95dc73599d77bd6e" is_pre_election: true
I20260925 15:22:50.062034 25299 raft_consensus.cc:2454] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e134e97e296f4cf2be9954dcf07613d2 in term 0.
I20260925 15:22:50.062178 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b712c53790ad4dab98e9607b55cc39d6" candidate_uuid: "e134e97e296f4cf2be9954dcf07613d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3361739e3142c59746d0bc39d8d1ec" is_pre_election: true
I20260925 15:22:50.062669 25562 raft_consensus.cc:2454] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e134e97e296f4cf2be9954dcf07613d2 in term 0.
I20260925 15:22:50.062707 25367 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e, e134e97e296f4cf2be9954dcf07613d2; no voters:
I20260925 15:22:50.063055 25630 raft_consensus.cc:2781] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:50.063200 25630 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:50.063324 25630 raft_consensus.cc:3037] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:50.065538 25630 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:50.066111 25630 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [CANDIDATE]: Term 1 election: Requested vote from peers 777078cf323148fe95dc73599d77bd6e (127.8.226.65:42451), 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693)
I20260925 15:22:50.066519 25299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b712c53790ad4dab98e9607b55cc39d6" candidate_uuid: "e134e97e296f4cf2be9954dcf07613d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777078cf323148fe95dc73599d77bd6e"
I20260925 15:22:50.066581 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b712c53790ad4dab98e9607b55cc39d6" candidate_uuid: "e134e97e296f4cf2be9954dcf07613d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3361739e3142c59746d0bc39d8d1ec"
I20260925 15:22:50.066691 25299 raft_consensus.cc:3037] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:50.066824 25562 raft_consensus.cc:3037] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:50.068852 25299 raft_consensus.cc:2454] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e134e97e296f4cf2be9954dcf07613d2 in term 1.
I20260925 15:22:50.069078 25562 raft_consensus.cc:2454] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e134e97e296f4cf2be9954dcf07613d2 in term 1.
I20260925 15:22:50.069280 25367 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e, e134e97e296f4cf2be9954dcf07613d2; no voters:
I20260925 15:22:50.069542 25630 raft_consensus.cc:2781] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:50.069965 25630 raft_consensus.cc:686] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 1 LEADER]: Becoming Leader. State: Replica: e134e97e296f4cf2be9954dcf07613d2, State: Running, Role: LEADER
I20260925 15:22:50.070451 25630 consensus_queue.cc:237] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:22:50.074752 25159 catalog_manager.cc:5495] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "e134e97e296f4cf2be9954dcf07613d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } health_report { overall_health: HEALTHY } } }
W20260925 15:22:50.104259 25608 tablet.cc:2369] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:50.124136 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.119134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.8.226.66:38837)
0925 15:22:50.119319 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:50.119323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:50.119340 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:22:50.119500 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:50.119504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:50.119579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:50.119699 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:50.120018 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.120814 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.122238 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.122616 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.122704 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.123126 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:50.123669 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:50.123776 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:50.123873 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:22:50.123933 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:22:50.132203 25299 raft_consensus.cc:1261] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Refusing update from remote peer e134e97e296f4cf2be9954dcf07613d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:50.132215 25562 raft_consensus.cc:1261] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Refusing update from remote peer e134e97e296f4cf2be9954dcf07613d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:50.133127 25634 consensus_queue.cc:1035] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:50.133490 25630 consensus_queue.cc:1035] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:50.158210 25159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:50.158510 25159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:50.161881 25159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Sending DeleteTablet for 3 replicas of tablet b712c53790ad4dab98e9607b55cc39d6
I20260925 15:22:50.162963 25411 tablet_service.cc:1514] Processing DeleteTablet for tablet b712c53790ad4dab98e9607b55cc39d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:50 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:37504
I20260925 15:22:50.163079 25279 tablet_service.cc:1514] Processing DeleteTablet for tablet b712c53790ad4dab98e9607b55cc39d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:50 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:52680
I20260925 15:22:50.163199 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet b712c53790ad4dab98e9607b55cc39d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:50 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:35992
I20260925 15:22:50.164052 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25196
I20260925 15:22:50.175911 25648 tablet_replica.cc:330] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: stopping tablet replica
I20260925 15:22:50.176294 25649 tablet_replica.cc:330] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: stopping tablet replica
I20260925 15:22:50.176805 25649 raft_consensus.cc:2229] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:50.177327 25648 raft_consensus.cc:2229] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:50.177397 25649 raft_consensus.cc:2258] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:50.178169 25648 raft_consensus.cc:2258] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:50.178190 25647 tablet_replica.cc:330] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: stopping tablet replica
I20260925 15:22:50.178586 25647 raft_consensus.cc:2229] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:22:50.179344 25647 raft_consensus.cc:2258] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:50.180614 25648 ts_tablet_manager.cc:1905] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:50.181280 25647 ts_tablet_manager.cc:1905] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:50.182382 25649 ts_tablet_manager.cc:1905] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:50.184655 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.165565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.8.226.65:42451)
0925 15:22:50.165743 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:50.165749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:50.165767 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:22:50.165817 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:50.165822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:50.165893 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:50.165988 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:50.167835 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.169033 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.170972 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.171610 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.171726 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.175796 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:50.175813 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:50.175831 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:50.175881 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:50.179980 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:50.180851 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:50.180859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:50.180862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:50.181051 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:50.182597 (+ 1546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:50.182603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:50.182606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:50.183208 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:50.183496 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:50.183988 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 15:22:50.184286 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":59}
I20260925 15:22:50.185635 25339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:50.189440 25647 ts_tablet_manager.cc:1918] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:50.189592 25648 ts_tablet_manager.cc:1918] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:50.189754 25648 log.cc:1196] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/b712c53790ad4dab98e9607b55cc39d6
I20260925 15:22:50.189769 25647 log.cc:1196] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/b712c53790ad4dab98e9607b55cc39d6
I20260925 15:22:50.190248 25648 ts_tablet_manager.cc:1939] T b712c53790ad4dab98e9607b55cc39d6 P 2b3361739e3142c59746d0bc39d8d1ec: Deleting consensus metadata
I20260925 15:22:50.190557 25647 ts_tablet_manager.cc:1939] T b712c53790ad4dab98e9607b55cc39d6 P e134e97e296f4cf2be9954dcf07613d2: Deleting consensus metadata
I20260925 15:22:50.192209 25146 catalog_manager.cc:4841] TS 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): tablet b712c53790ad4dab98e9607b55cc39d6 (table test-table [id=39d7cbfcdfc443f0abe3a901abc34af0]) successfully deleted
I20260925 15:22:50.192745 25144 catalog_manager.cc:4841] TS e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): tablet b712c53790ad4dab98e9607b55cc39d6 (table test-table [id=39d7cbfcdfc443f0abe3a901abc34af0]) successfully deleted
I20260925 15:22:50.276194 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 25196
W20260925 15:22:50.300623 25146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:42451 (error 108)
I20260925 15:22:50.300889 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25348
I20260925 15:22:50.302811 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.302338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52690)
0925 15:22:50.302531 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.302647 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260925 15:22:50.303083 25146 catalog_manager.cc:4584] TS 777078cf323148fe95dc73599d77bd6e (127.8.226.65:42451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b712c53790ad4dab98e9607b55cc39d6: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
I20260925 15:22:50.318785 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.301994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.8.226.66:38837)
0925 15:22:50.302301 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:50.302306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:50.302321 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:22:50.302395 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:50.302402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:50.302488 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:50.302588 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:50.304334 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.305290 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.307132 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.307541 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.307637 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.310957 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:50.310971 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:50.310990 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:50.311048 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:50.315058 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:50.315927 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:50.315954 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:50.315957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:50.316143 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:50.316594 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:50.316601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:50.316605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:50.317108 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:50.317472 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:50.318166 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 15:22:50.318433 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 15:22:50.319667 25471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:50.356845 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.356370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52702)
0925 15:22:50.356588 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.356684 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 15:22:50.357838 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.357498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52718)
0925 15:22:50.357635 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.357726 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:22:50.418342 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.417905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52722)
0925 15:22:50.418110 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.418195 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:22:50.419334 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.419003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52730)
0925 15:22:50.419150 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.419227 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:22:50.431881 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 25348
I20260925 15:22:50.450114 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25479
I20260925 15:22:50.463732 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.450842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.8.226.67:38693)
0925 15:22:50.451012 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:50.451018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:50.451064 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:22:50.451296 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:50.451302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:50.451378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:50.451485 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:50.452850 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.453707 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.454969 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:50.455378 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:50.455505 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:50.458221 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:50.458237 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:50.458252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:50.458288 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:50.461025 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:50.461694 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:50.461704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:50.461707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:50.461857 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:50.462223 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:50.462230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:50.462234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:50.462578 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:50.462882 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:50.463171 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:22:50.463383 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:22:50.464347 25602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:50.499724 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.499276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52734)
0925 15:22:50.499475 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.499564 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:22:50.500777 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.500417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52742)
0925 15:22:50.500590 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.500674 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 15:22:50.557087 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 25479
I20260925 15:22:50.573716 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25126
I20260925 15:22:50.574867 25189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:22:50.688848 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 25126
I20260925 15:22:50.689002 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.688237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:42451 (local address 127.0.0.1:52758)
0925 15:22:50.688674 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:50.688773 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
I20260925 15:23:10.710040 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44275
--webserver_interface=127.8.226.126
--webserver_port=38951
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:44275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:11.044723 25655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:11.045086 25655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:11.045243 25655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:11.052568 25655 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:23:11.052695 25655 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:23:11.052762 25655 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:23:11.052834 25655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:11.052909 25655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:11.053025 25655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:11.053107 25655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:11.053192 25655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:11.063264 25655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:44275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:44275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=38951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:11.064572 25655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:11.065991 25655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:11.077903 25661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:11.077926 25663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:11.078553 25655 server_base.cc:1034] running on GCE node
W20260925 15:23:11.078590 25660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:11.079450 25655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:11.080875 25655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:11.082141 25655 hybrid_clock.cc:648] HybridClock initialized: now 1790349791082076 us; error 36 us; skew 500 ppm
I20260925 15:23:11.082475 25655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:11 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349791, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:11.085855 25655 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:23:11.087307 25655 webserver.cc:460] Webserver started at http://127.8.226.126:38951/ using document root <none> and password file <none>
I20260925 15:23:11.087983 25655 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:11.088109 25655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:11.093555 25655 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:23:11.096328 25670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:11.097678 25655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:11.097870 25655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "49de814a44a9452385f7881cbfea51db"
format_stamp: "Formatted at 2026-09-25 15:22:25 on dist-test-slave-8vw7"
server_key: "fa038b25dd870fd93fc7a90cc5bd0ba4"
server_key_iv: "2794ec83e05af9c559a30985defed4c4"
server_key_version: "encryptionkey@0"
I20260925 15:23:11.098815 25655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:11.114276 25655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:11.118568 25655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:11.118852 25655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:11.137336 25655 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:44275
I20260925 15:23:11.137360 25721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:44275 every 8 connection(s)
I20260925 15:23:11.138875 25655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:11.145617 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25655
I20260925 15:23:11.145641 25722 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:23:11.146417 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:42451
--local_ip_for_outbound_sockets=127.8.226.65
--tserver_master_addrs=127.8.226.126:44275
--webserver_port=43265
--webserver_interface=127.8.226.65
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:23:11.148877 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap starting.
I20260925 15:23:11.176286 25722 log.cc:824] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:11.197402 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:11.198064 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Bootstrap complete.
I20260925 15:23:11.212208 25722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:23:11.213847 25722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49de814a44a9452385f7881cbfea51db, State: Initialized, Role: FOLLOWER
I20260925 15:23:11.214751 25722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:23:11.215126 25722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:11.215281 25722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:11.215507 25722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:23:11.221479 25722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:23:11.222062 25722 leader_election.cc:302] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49de814a44a9452385f7881cbfea51db; no voters:
I20260925 15:23:11.222798 25722 leader_election.cc:288] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:23:11.222975 25726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:23:11.224283 25726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [term 3 LEADER]: Becoming Leader. State: Replica: 49de814a44a9452385f7881cbfea51db, State: Running, Role: LEADER
I20260925 15:23:11.224920 25722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:11.224843 25726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } }
I20260925 15:23:11.227259 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "49de814a44a9452385f7881cbfea51db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } } }
I20260925 15:23:11.227596 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:11.231698 25728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49de814a44a9452385f7881cbfea51db. Latest consensus state: current_term: 3 leader_uuid: "49de814a44a9452385f7881cbfea51db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49de814a44a9452385f7881cbfea51db" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 44275 } } }
I20260925 15:23:11.232743 25728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:11.239914 25731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:11.248478 25731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:11.250041 25731 catalog_manager.cc:1259] Loaded cluster ID: 0beff253563b471da978748d5240075d
I20260925 15:23:11.250201 25731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:11.254417 25731 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:11.256173 25731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Loaded TSK: 0
I20260925 15:23:11.256306 25731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49de814a44a9452385f7881cbfea51db: Loaded TSK: 1
F20260925 15:23:11.257465 25731 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f075cd419ad google::LogMessage::Fail() at ??:0
@ 0x7f075cd4488a google::LogMessage::SendToLog() at ??:0
@ 0x7f075cd414de google::LogMessage::Flush() at ??:0
@ 0x7f075cd42a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0760aa57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0760a9473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f07727701c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f077276f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f07727fb2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f07727fb0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f07728455e9 std::function<>::operator()() at ??:0
@ 0x7f0772773cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0772771c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f07727f9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f07727f9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f075f98d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f075f99091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f075f990799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f075f96beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0762c456db start_thread at ??:0
@ 0x7f075bbdb61f clone at ??:0
W20260925 15:23:11.570315 25724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:11.570588 25724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:11.570778 25724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:11.577894 25724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:11.578037 25724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:11.578243 25724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:11.587704 25724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:42451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=43265
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:11.589039 25724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:11.590368 25724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:11.603503 25772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:11.603533 25770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:11.603956 25769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:11.604295 25724 server_base.cc:1034] running on GCE node
I20260925 15:23:11.604907 25724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:11.605870 25724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:11.607097 25724 hybrid_clock.cc:648] HybridClock initialized: now 1790349791607063 us; error 51 us; skew 500 ppm
I20260925 15:23:11.607405 25724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:11 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349791, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:11.610468 25724 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:23:11.611896 25724 webserver.cc:460] Webserver started at http://127.8.226.65:43265/ using document root <none> and password file <none>
I20260925 15:23:11.612617 25724 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:11.612782 25724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:11.618861 25724 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:23:11.621876 25779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:11.623202 25724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:23:11.623371 25724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "777078cf323148fe95dc73599d77bd6e"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "1c41aa650c7e7210913f7adcac0eb217"
server_key_iv: "d518cc3378d08c4942500fc7018c32a2"
server_key_version: "encryptionkey@0"
I20260925 15:23:11.624347 25724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:11.638233 25724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:11.642036 25724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:11.642305 25724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:11.643230 25724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:11.645056 25786 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 15:23:11.650722 25724 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:23:11.650856 25724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 15:23:11.650990 25724 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:23:11.654115 25724 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:23:11.654220 25724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:11.654635 25786 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Bootstrap starting.
I20260925 15:23:11.663159 25786 log.cc:824] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:11.678222 25786 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:11.679064 25786 tablet_bootstrap.cc:492] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Bootstrap complete.
I20260925 15:23:11.679726 25786 ts_tablet_manager.cc:1397] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.004s
I20260925 15:23:11.690948 25786 raft_consensus.cc:348] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:11.692572 25786 raft_consensus.cc:729] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777078cf323148fe95dc73599d77bd6e, State: Initialized, Role: FOLLOWER
I20260925 15:23:11.693378 25786 consensus_queue.cc:260] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:11.694756 25724 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:42451
I20260925 15:23:11.694784 25894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:42451 every 8 connection(s)
I20260925 15:23:11.695981 25724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:11.696980 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25724
I20260925 15:23:11.697932 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:38837
--local_ip_for_outbound_sockets=127.8.226.66
--tserver_master_addrs=127.8.226.126:44275
--webserver_port=36421
--webserver_interface=127.8.226.66
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:23:11.701771 25786 ts_tablet_manager.cc:1428] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260925 15:23:11.703899 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:11.701960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:34763)
0925 15:23:11.702841 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:11.703084 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":544,"threads_started":1}
W20260925 15:23:11.704563 25895 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:44275 (0 consecutive failures): Network error: Failed to ping master at 127.8.226.126:44275: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
I20260925 15:23:11.705456 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:11.705159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:39559)
0925 15:23:11.705296 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:11.705360 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:23:11.706430 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:11.705972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:43307)
0925 15:23:11.706167 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:11.706273 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 15:23:11.706710 25895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:23:12.023348 25898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:12.023690 25898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:12.023891 25898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:12.031697 25898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:12.031868 25898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:12.032045 25898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:12.041785 25898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:38837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=36421
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:12.043128 25898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:12.044432 25898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:12.057842 25903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:12.057734 25904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:12.057782 25906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:12.058518 25898 server_base.cc:1034] running on GCE node
I20260925 15:23:12.059103 25898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:12.060065 25898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:12.061322 25898 hybrid_clock.cc:648] HybridClock initialized: now 1790349792061298 us; error 62 us; skew 500 ppm
I20260925 15:23:12.061686 25898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:12 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:12.064814 25898 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:23:12.066329 25898 webserver.cc:460] Webserver started at http://127.8.226.66:36421/ using document root <none> and password file <none>
I20260925 15:23:12.067003 25898 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:12.067147 25898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:12.072278 25898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:12.074965 25913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:12.076187 25898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:12.076362 25898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "e134e97e296f4cf2be9954dcf07613d2"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "ce6c710d113a624443f42d4c83f233ee"
server_key_iv: "4d86f66f7354ba368924eb8789aa0361"
server_key_version: "encryptionkey@0"
I20260925 15:23:12.077382 25898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:12.097620 25898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:12.101325 25898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:12.101601 25898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:12.102540 25898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:12.103813 25898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:12.103937 25898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:12.104074 25898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:12.104159 25898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:12.137053 25898 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:38837
I20260925 15:23:12.137151 26025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:38837 every 8 connection(s)
I20260925 15:23:12.138479 25898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:12.143128 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.141376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:60695)
0925 15:23:12.142064 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.142291 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":257,"threads_started":1}
W20260925 15:23:12.143879 26026 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:44275 (0 consecutive failures): Network error: Failed to ping master at 127.8.226.126:44275: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
I20260925 15:23:12.144619 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.144320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:40089)
0925 15:23:12.144452 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.144523 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:23:12.145498 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 25898
I20260925 15:23:12.145991 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.145669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:44787)
0925 15:23:12.145801 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.145874 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:23:12.146211 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:38693
--local_ip_for_outbound_sockets=127.8.226.67
--tserver_master_addrs=127.8.226.126:44275
--webserver_port=34151
--webserver_interface=127.8.226.67
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:12.146263 26026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:23:12.469869 26029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:12.470129 26029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:12.470311 26029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:12.477375 26029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:12.477520 26029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:12.477712 26029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:12.487226 26029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:38693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=34151
--tserver_master_addrs=127.8.226.126:44275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:12.488435 26029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:12.489774 26029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:12.502336 26034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:12.502378 26037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:12.502426 26035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:12.503322 26029 server_base.cc:1034] running on GCE node
I20260925 15:23:12.503927 26029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:12.504814 26029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:12.506011 26029 hybrid_clock.cc:648] HybridClock initialized: now 1790349792505952 us; error 57 us; skew 500 ppm
I20260925 15:23:12.506310 26029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:12 dist-test-slave-8vw7 krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:12.509348 26029 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:23:12.510668 26029 webserver.cc:460] Webserver started at http://127.8.226.67:34151/ using document root <none> and password file <none>
I20260925 15:23:12.511307 26029 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:12.511444 26029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:12.516397 26029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:23:12.518992 26044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:12.520226 26029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:12.520385 26029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "2b3361739e3142c59746d0bc39d8d1ec"
format_stamp: "Formatted at 2026-09-25 15:22:26 on dist-test-slave-8vw7"
server_key: "34687d96c6f5a04fe16e855f091e97f2"
server_key_iv: "1de19a7df32fa2685420bda949f76460"
server_key_version: "encryptionkey@0"
I20260925 15:23:12.521369 26029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:12.544065 26029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:12.547700 26029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:12.547950 26029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:12.548892 26029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:12.550212 26029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:12.550333 26029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:12.550443 26029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:12.550525 26029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:12.582618 26029 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:38693
I20260925 15:23:12.582695 26156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:38693 every 8 connection(s)
I20260925 15:23:12.583968 26029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:12.588456 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.586734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:59753)
0925 15:23:12.587428 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.587652 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260925 15:23:12.589273 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26029
W20260925 15:23:12.589301 26157 heartbeater.cc:644] Failed to heartbeat to 127.8.226.126:44275 (0 consecutive failures): Network error: Failed to ping master at 127.8.226.126:44275: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
I20260925 15:23:12.590096 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.589792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:47705)
0925 15:23:12.589934 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.590007 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:23:12.590902 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.590597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:55779)
0925 15:23:12.590725 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.590805 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:23:12.591122 26157 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:23:12.707787 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.707145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:48831)
0925 15:23:12.707525 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.707599 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":184,"threads_started":1}
I20260925 15:23:13.147622 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.146834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:53391)
0925 15:23:13.147301 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.147393 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":231,"threads_started":1}
I20260925 15:23:13.152902 26163 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:13.153539 26163 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:13.155128 26163 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693), e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
Sep 25 15:23:13 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349791, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
Sep 25 15:23:13 dist-test-slave-8vw7 krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349791, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:23:13.183884 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.155473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:38837 (local address 127.8.226.65:46981)
0925 15:23:13.155845 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.155896 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:13.156598 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:13.161206 (+ 4608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:13.161218 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:13.162250 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:13.163506 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:13.163536 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.166021 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.166025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:13.168564 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:13.168581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.170425 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.170433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:13.170816 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:13.171937 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:13.171992 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:13.176112 (+ 4120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:13.179966 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:13.179973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:13.179989 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:13.180650 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:13.181265 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:13.181268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:13.181270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:13.181468 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:13.182327 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:13.182336 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:13.182659 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 15:23:13.182926 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 15:23:13.183443 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":502,"spinlock_wait_cycles":125440}
I20260925 15:23:13.183954 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.155799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:48307 (local address 127.8.226.67:38693)
0925 15:23:13.157763 (+ 1964us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:13.157785 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:13.157831 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:23:13.160056 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:13.160069 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:13.160585 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:13.161022 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:13.163801 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.165783 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.168751 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.170110 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.170320 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:13.176310 (+ 5990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:13.176318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:13.176337 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:13.176379 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:13.179772 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:13.180815 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:13.180822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:13.180829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:13.181070 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:13.181632 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:13.181636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:13.181638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:13.182190 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:13.182387 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:13.183170 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 15:23:13.183560 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1777,"thread_start_us":1670,"threads_started":1}
I20260925 15:23:13.183971 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.155787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:46981 (local address 127.8.226.66:38837)
0925 15:23:13.158123 (+ 2336us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:13.158135 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:13.158178 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:23:13.160375 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:13.160385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:13.160787 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:13.161201 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:13.163807 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.165783 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.168751 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.170110 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.170326 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:13.176434 (+ 6108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:13.176442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:13.176456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:13.176501 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:13.179787 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:13.180814 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:13.180822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:13.180829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:13.181101 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:13.181585 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:13.181591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:13.181595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:13.182160 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:13.182358 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:13.183168 (+ 810us) server_negotiation.cc:299] Negotiation successful
0925 15:23:13.183595 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165,"thread_start_us":2018,"threads_started":1}
I20260925 15:23:13.185128 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.155472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:38693 (local address 127.8.226.65:48307)
0925 15:23:13.157554 (+ 2082us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.157577 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:13.157704 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:13.161206 (+ 3502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:13.161218 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:13.162250 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:13.163506 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:13.163536 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.166019 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.166025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:13.168568 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:13.168581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:13.170435 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:13.170441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:13.170817 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:13.171937 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:13.171972 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:13.175925 (+ 3953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:13.179953 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:13.179964 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:13.179981 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:13.180647 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:13.181269 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:13.181275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:13.181279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:13.181476 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:13.182357 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:13.182362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:13.182648 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 15:23:13.182927 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 15:23:13.184713 (+ 1786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1895,"mutex_wait_us":631,"spinlock_wait_cycles":29184,"thread_start_us":1481,"threads_started":1}
W20260925 15:23:13.187768 25783 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
W20260925 15:23:13.188014 25781 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
I20260925 15:23:13.188143 25781 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e; no voters: 2b3361739e3142c59746d0bc39d8d1ec, e134e97e296f4cf2be9954dcf07613d2
I20260925 15:23:13.188465 26163 raft_consensus.cc:2726] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:23:13.592525 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.591593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:58221)
0925 15:23:13.592103 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.592300 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
I20260925 15:23:13.709263 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.708571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:43959)
0925 15:23:13.709014 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.709103 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":211,"threads_started":1}
I20260925 15:23:14.149158 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:14.148442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:56495)
0925 15:23:14.148861 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:14.148937 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
I20260925 15:23:14.594013 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:14.593353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:60859)
0925 15:23:14.593753 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:14.593827 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":215,"threads_started":1}
I20260925 15:23:14.710773 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:14.710096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:54153)
0925 15:23:14.710526 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:14.710603 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260925 15:23:15.150681 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:15.149943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:42139)
0925 15:23:15.150353 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:15.150467 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":226,"threads_started":1}
I20260925 15:23:15.374583 26173 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:15.374825 26173 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:15.375460 26173 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693), e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
W20260925 15:23:15.376628 25783 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
W20260925 15:23:15.376847 25781 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
I20260925 15:23:15.376942 25781 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e; no voters: 2b3361739e3142c59746d0bc39d8d1ec, e134e97e296f4cf2be9954dcf07613d2
I20260925 15:23:15.377228 26173 raft_consensus.cc:2726] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:23:15.595512 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:15.594853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:59697)
0925 15:23:15.595278 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:15.595356 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
I20260925 15:23:15.712409 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:15.711627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:55827)
0925 15:23:15.712128 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:15.712229 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":221,"threads_started":1}
I20260925 15:23:16.152115 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.151460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:33365)
0925 15:23:16.151863 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.151940 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260925 15:23:16.597426 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.596499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:42159)
0925 15:23:16.597099 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.597218 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260925 15:23:16.714143 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.713456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:60597)
0925 15:23:16.713898 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.713971 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":211,"threads_started":1}
I20260925 15:23:17.153700 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.153069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:33305)
0925 15:23:17.153470 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:17.153543 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260925 15:23:17.598874 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.598237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:38569)
0925 15:23:17.598623 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:17.598696 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":213,"threads_started":1}
I20260925 15:23:17.715642 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.715007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:41483)
0925 15:23:17.715418 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:17.715495 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":206,"threads_started":1}
I20260925 15:23:17.901878 26182 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:17.902045 26182 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:17.902594 26182 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693), e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
W20260925 15:23:17.903610 25781 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
W20260925 15:23:17.903812 25783 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
I20260925 15:23:17.903915 25783 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e; no voters: 2b3361739e3142c59746d0bc39d8d1ec, e134e97e296f4cf2be9954dcf07613d2
I20260925 15:23:17.904150 26182 raft_consensus.cc:2726] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:23:18.155135 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:18.154483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:53227)
0925 15:23:18.154893 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:18.154968 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260925 15:23:18.600601 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:18.599907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:57021)
0925 15:23:18.600359 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:18.600436 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260925 15:23:18.717173 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:18.716447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:48791)
0925 15:23:18.716824 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:18.716908 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":200,"threads_started":1}
I20260925 15:23:19.156777 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:19.156019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:33245)
0925 15:23:19.156500 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:19.156586 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":222,"threads_started":1}
I20260925 15:23:19.602319 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:19.601587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:48523)
0925 15:23:19.602059 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:19.602144 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":220,"threads_started":1}
I20260925 15:23:19.718684 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:19.718014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:35887)
0925 15:23:19.718435 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:19.718523 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260925 15:23:20.158511 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:20.157796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:43033)
0925 15:23:20.158247 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:20.158345 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260925 15:23:20.603873 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:20.603179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:40213)
0925 15:23:20.603588 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:20.603665 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":223,"threads_started":1}
I20260925 15:23:20.720325 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:20.719495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:36815)
0925 15:23:20.719990 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:20.720069 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":241,"threads_started":1}
I20260925 15:23:21.160051 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:21.159362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:33427)
0925 15:23:21.159817 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:21.159900 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":219,"threads_started":1}
I20260925 15:23:21.605661 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:21.604827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:56143)
0925 15:23:21.605372 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:21.605456 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":297,"threads_started":1}
I20260925 15:23:21.722077 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:21.721415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:39597)
0925 15:23:21.721853 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:21.721930 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":246,"threads_started":1}
I20260925 15:23:22.161777 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:22.160994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:41643)
0925 15:23:22.161525 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:22.161602 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":298,"threads_started":1}
I20260925 15:23:22.607193 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:22.606492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:52405)
0925 15:23:22.606925 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:22.607008 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
I20260925 15:23:22.663466 26197 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:22.663692 26197 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:22.664323 26197 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693), e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
W20260925 15:23:22.665318 25781 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
W20260925 15:23:22.665504 25783 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
I20260925 15:23:22.665617 25783 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e; no voters: 2b3361739e3142c59746d0bc39d8d1ec, e134e97e296f4cf2be9954dcf07613d2
I20260925 15:23:22.665843 26197 raft_consensus.cc:2726] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:23:22.723737 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:22.722920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:41351)
0925 15:23:22.723404 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:22.723522 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20260925 15:23:23.163245 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:23.162560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:56925)
0925 15:23:23.162995 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:23.163069 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260925 15:23:23.608636 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:23.607991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:41073)
0925 15:23:23.608406 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:23.608487 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":206,"threads_started":1}
I20260925 15:23:23.725366 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:23.724602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:45473)
0925 15:23:23.725083 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:23.725162 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":228,"threads_started":1}
I20260925 15:23:24.164840 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:24.164194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:36209)
0925 15:23:24.164591 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:24.164676 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":216,"threads_started":1}
I20260925 15:23:24.199447 26203 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:24.199622 26203 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:24.200186 26203 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693), e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
W20260925 15:23:24.201253 25783 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
W20260925 15:23:24.201455 25781 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
I20260925 15:23:24.201607 25781 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e; no voters: 2b3361739e3142c59746d0bc39d8d1ec, e134e97e296f4cf2be9954dcf07613d2
I20260925 15:23:24.201869 26203 raft_consensus.cc:2726] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:23:24.610149 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:24.609459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:44145)
0925 15:23:24.609904 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:24.609984 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260925 15:23:24.726909 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:24.726227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:42759)
0925 15:23:24.726649 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:24.726738 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260925 15:23:25.166527 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:25.165807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:34377)
0925 15:23:25.166284 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:25.166378 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":275,"threads_started":1}
I20260925 15:23:25.611943 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:25.611121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:48899)
0925 15:23:25.611640 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:25.611743 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260925 15:23:25.728565 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:25.727724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:45265)
0925 15:23:25.728297 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:25.728393 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":318,"threads_started":1}
I20260925 15:23:26.168164 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:26.167420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:56735)
0925 15:23:26.167889 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:26.167962 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":251,"threads_started":1}
I20260925 15:23:26.613643 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:26.612802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:53483)
0925 15:23:26.613376 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:26.613456 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":376,"threads_started":1}
I20260925 15:23:26.730472 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:26.729767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:51203)
0925 15:23:26.730222 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:26.730310 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260925 15:23:27.072145 26212 raft_consensus.cc:482] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:27.072392 26212 raft_consensus.cc:504] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777078cf323148fe95dc73599d77bd6e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 42451 } } peers { permanent_uuid: "2b3361739e3142c59746d0bc39d8d1ec" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 38693 } } peers { permanent_uuid: "e134e97e296f4cf2be9954dcf07613d2" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 38837 } }
I20260925 15:23:27.073107 26212 leader_election.cc:288] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693), e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837)
W20260925 15:23:27.074105 25781 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e134e97e296f4cf2be9954dcf07613d2 (127.8.226.66:38837): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
W20260925 15:23:27.074309 25783 leader_election.cc:341] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b3361739e3142c59746d0bc39d8d1ec (127.8.226.67:38693): Not found: Tablet not found: b712c53790ad4dab98e9607b55cc39d6
I20260925 15:23:27.074456 25783 leader_election.cc:302] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 777078cf323148fe95dc73599d77bd6e; no voters: 2b3361739e3142c59746d0bc39d8d1ec, e134e97e296f4cf2be9954dcf07613d2
I20260925 15:23:27.074704 26212 raft_consensus.cc:2726] T b712c53790ad4dab98e9607b55cc39d6 P 777078cf323148fe95dc73599d77bd6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:23:27.169965 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:27.169250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:57383)
0925 15:23:27.169714 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:27.169802 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260925 15:23:27.601647 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25724
I20260925 15:23:27.615384 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:27.614669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:59173)
0925 15:23:27.615112 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:27.615232 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":238,"threads_started":1}
I20260925 15:23:27.732261 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:27.731506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.65:40961)
0925 15:23:27.731965 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:27.732057 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
I20260925 15:23:28.171804 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:28.171033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.66:60645)
0925 15:23:28.171516 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:28.171606 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":255,"threads_started":1}
I20260925 15:23:28.523900 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:27.602881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.8.226.65:42451)
0925 15:23:27.603531 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:27.603543 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:28.501796 (+898253us) server_negotiation.cc:407] Connection header received
0925 15:23:28.501989 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:28.501997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:28.502243 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:28.502431 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:28.503922 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:28.504883 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:28.506192 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:28.506802 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:28.506946 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:28.517798 (+ 10852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:28.517816 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:28.517832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:28.517875 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:28.520782 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:28.521625 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:28.521633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:28.521637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:28.521780 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:28.522210 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:28.522214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:28.522216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:28.522592 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:28.522911 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:28.523264 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 15:23:28.523545 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":276,"threads_started":1}
I20260925 15:23:28.525372 25889 generic_service.cc:196] Checking for leaks (request via RPC)
W20260925 15:23:28.603627 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25724: Timed out: CheckLeaks RPC to 127.8.226.65:42451 timed out after 1.000s (SENT)
I20260925 15:23:28.603899 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 25724
I20260925 15:23:28.617296 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:28.616467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:44275 (local address 127.8.226.67:58663)
0925 15:23:28.617029 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:28.617122 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.126:44275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":255,"threads_started":1}
I20260925 15:23:28.621793 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 25898
I20260925 15:23:28.635360 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:28.622478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.8.226.66:38837)
0925 15:23:28.623009 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:28.623014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:28.623029 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:23:28.623089 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:28.623093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:28.623149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:28.623275 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:28.624668 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:28.625510 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:28.626766 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:28.627213 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:28.627352 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:28.630073 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:28.630078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:28.630081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:28.630114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:28.632663 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:28.633389 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:28.633395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:28.633399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:28.633533 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:28.633900 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:28.633905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:28.633909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:28.634181 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:28.634472 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:28.634755 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:23:28.634979 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":250,"threads_started":1}
I20260925 15:23:28.636082 26021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:28.720149 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 25898
I20260925 15:23:28.737679 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26029
I20260925 15:23:28.751400 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:28.738492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.8.226.67:38693)
0925 15:23:28.739039 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:28.739044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:28.739058 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:23:28.739111 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:28.739115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:28.739170 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:28.739307 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:28.740687 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:28.741542 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:28.742860 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:28.743280 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:28.743409 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:28.746015 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:28.746020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:28.746023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:28.746054 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:28.748741 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:28.749449 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:28.749456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:28.749460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:28.749602 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:28.749987 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:28.749993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:28.749997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:28.750258 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:28.750546 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:28.750796 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:23:28.751014 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":304,"threads_started":1}
I20260925 15:23:28.752177 26152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:28.837738 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26029
2026-09-25T15:23:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65083 ms)
[----------] 30 tests from SecurityITest (285613 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:28 dist-test-slave-8vw7 krb5kdc[26226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:28 dist-test-slave-8vw7 krb5kdc[26226](info): set up 2 sockets
Sep 25 15:23:28 dist-test-slave-8vw7 krb5kdc[26226](info): commencing operation
krb5kdc: starting...
W20260925 15:23:29.953325 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:30 dist-test-slave-8vw7 krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:23:30.196072 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35563
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:35563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:30.520843 26239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:30.521200 26239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:30.521363 26239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:30.528650 26239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:30.528800 26239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:30.528914 26239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:30.529035 26239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:30.529110 26239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:30.538892 26239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:35563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:30.540221 26239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:30.541567 26239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:30.553064 26247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:30.553074 26245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:30.553283 26244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:30.553745 26239 server_base.cc:1034] running on GCE node
I20260925 15:23:30.554677 26239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:30.556074 26239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:30.557305 26239 hybrid_clock.cc:648] HybridClock initialized: now 1790349810557263 us; error 48 us; skew 500 ppm
I20260925 15:23:30.557646 26239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:30 dist-test-slave-8vw7 krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349810, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:30.560606 26239 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:23:30.562124 26239 webserver.cc:460] Webserver started at http://127.8.226.126:39865/ using document root <none> and password file <none>
I20260925 15:23:30.562805 26239 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:30.562945 26239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:30.563257 26239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:30.566206 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "b09f1def95d0481fb7de0a392b878778"
format_stamp: "Formatted at 2026-09-25 15:23:30 on dist-test-slave-8vw7"
server_key: "1751b2c42850e8b66aa3d994ca455dd1"
server_key_iv: "8056160099206c656622f979dccfa87d"
server_key_version: "encryptionkey@0"
I20260925 15:23:30.566942 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "b09f1def95d0481fb7de0a392b878778"
format_stamp: "Formatted at 2026-09-25 15:23:30 on dist-test-slave-8vw7"
server_key: "1751b2c42850e8b66aa3d994ca455dd1"
server_key_iv: "8056160099206c656622f979dccfa87d"
server_key_version: "encryptionkey@0"
I20260925 15:23:30.571830 26239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:23:30.575073 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:30.576344 26239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:23:30.576541 26239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "b09f1def95d0481fb7de0a392b878778"
format_stamp: "Formatted at 2026-09-25 15:23:30 on dist-test-slave-8vw7"
server_key: "1751b2c42850e8b66aa3d994ca455dd1"
server_key_iv: "8056160099206c656622f979dccfa87d"
server_key_version: "encryptionkey@0"
I20260925 15:23:30.576745 26239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:30.595022 26239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:30.598695 26239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:30.598973 26239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:30.617005 26239 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:35563
I20260925 15:23:30.617085 26305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:35563 every 8 connection(s)
I20260925 15:23:30.618322 26239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:30.618829 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26239
I20260925 15:23:30.619061 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:30.619863 9097 external_mini_cluster.cc:1428] Setting key 3d7b98ee027ac29c4089f3bee06f77fb
I20260925 15:23:30.622565 26306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:30.632543 26306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778: Bootstrap starting.
Sep 25 15:23:30 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:30.635583 26306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:30.636598 26306 log.cc:824] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:30.639585 26306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778: No bootstrap required, opened a new log
I20260925 15:23:30.640765 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:30.621773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.8.226.126:35563)
0925 15:23:30.622514 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:30.622526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:30.622566 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:23:30.624214 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:30.624234 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:30.624585 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:30.624991 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:30.626453 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:30.627749 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:30.629002 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:30.629675 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:30.629895 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:30.633889 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:30.633897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:30.633914 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:30.633955 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:30.637006 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:30.637763 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:30.637771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:30.637781 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:30.637938 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:30.638345 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:30.638348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:30.638350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:30.638796 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:30.638997 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:30.639464 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:23:30.639840 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":356,"threads_started":1}
I20260925 15:23:30.647396 26306 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09f1def95d0481fb7de0a392b878778" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35563 } }
I20260925 15:23:30.647735 26306 raft_consensus.cc:374] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:30.647845 26306 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f1def95d0481fb7de0a392b878778, State: Initialized, Role: FOLLOWER
I20260925 15:23:30.648370 26306 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09f1def95d0481fb7de0a392b878778" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35563 } }
I20260925 15:23:30.648587 26306 raft_consensus.cc:388] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:30.648747 26306 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:30.648905 26306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:30.651183 26306 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09f1def95d0481fb7de0a392b878778" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35563 } }
I20260925 15:23:30.651640 26306 leader_election.cc:302] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09f1def95d0481fb7de0a392b878778; no voters:
I20260925 15:23:30.652262 26306 leader_election.cc:288] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:30.652343 26311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:30.653076 26311 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [term 1 LEADER]: Becoming Leader. State: Replica: b09f1def95d0481fb7de0a392b878778, State: Running, Role: LEADER
I20260925 15:23:30.653621 26311 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09f1def95d0481fb7de0a392b878778" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35563 } }
I20260925 15:23:30.654448 26306 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:30.656623 26313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09f1def95d0481fb7de0a392b878778. Latest consensus state: current_term: 1 leader_uuid: "b09f1def95d0481fb7de0a392b878778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09f1def95d0481fb7de0a392b878778" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35563 } } }
I20260925 15:23:30.656699 26312 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09f1def95d0481fb7de0a392b878778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09f1def95d0481fb7de0a392b878778" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35563 } } }
I20260925 15:23:30.657078 26313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:30.657232 26312 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:30.666165 26318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:30.671416 26318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:30.680145 26318 catalog_manager.cc:1347] Generated new cluster ID: 61a69220d4044ebf8d837e2031862379
I20260925 15:23:30.680327 26318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:30.710035 26318 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:30.711946 26318 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:30.727128 26318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778: Generated new TSK 0
I20260925 15:23:30.728041 26318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:23:30.792235 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35563
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:31.121975 26332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:31.122270 26332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:31.122427 26332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:31.129840 26332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:31.129953 26332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:31.130141 26332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:31.140112 26332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:31.141554 26332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:31.142872 26332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:31.155999 26337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:31.156086 26340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:31.156044 26338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:31.156855 26332 server_base.cc:1034] running on GCE node
I20260925 15:23:31.157498 26332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:31.158464 26332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:31.159724 26332 hybrid_clock.cc:648] HybridClock initialized: now 1790349811159652 us; error 62 us; skew 500 ppm
I20260925 15:23:31.160066 26332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:31 dist-test-slave-8vw7 krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:31.163331 26332 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:23:31.164698 26332 webserver.cc:460] Webserver started at http://127.8.226.65:38747/ using document root <none> and password file <none>
I20260925 15:23:31.165418 26332 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:31.165519 26332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:31.165786 26332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:31.168795 26332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "62f481f4369b4e7fafccbfc2be41e367"
format_stamp: "Formatted at 2026-09-25 15:23:31 on dist-test-slave-8vw7"
server_key: "2654128fdef19b34f6f48f14c32a0255"
server_key_iv: "5cc669c514e52a4cc1ed28f79a57b86a"
server_key_version: "encryptionkey@0"
I20260925 15:23:31.169644 26332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "62f481f4369b4e7fafccbfc2be41e367"
format_stamp: "Formatted at 2026-09-25 15:23:31 on dist-test-slave-8vw7"
server_key: "2654128fdef19b34f6f48f14c32a0255"
server_key_iv: "5cc669c514e52a4cc1ed28f79a57b86a"
server_key_version: "encryptionkey@0"
I20260925 15:23:31.174667 26332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:23:31.178011 26347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:31.179368 26332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:23:31.179499 26332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "62f481f4369b4e7fafccbfc2be41e367"
format_stamp: "Formatted at 2026-09-25 15:23:31 on dist-test-slave-8vw7"
server_key: "2654128fdef19b34f6f48f14c32a0255"
server_key_iv: "5cc669c514e52a4cc1ed28f79a57b86a"
server_key_version: "encryptionkey@0"
I20260925 15:23:31.179670 26332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:31.197682 26332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:31.201683 26332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:31.201936 26332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:31.202840 26332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:31.204201 26332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:31.204274 26332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:31.204357 26332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:31.204401 26332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:31.238128 26332 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:34053
I20260925 15:23:31.238217 26459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:34053 every 8 connection(s)
I20260925 15:23:31.239461 26332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:31.245820 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26332
I20260925 15:23:31.246018 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:31.246775 9097 external_mini_cluster.cc:1428] Setting key 0c7e38a5f4dbb11edcdea53ee900287f
Sep 25 15:23:31 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:31.260470 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:31.241961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:33649 (local address 127.8.226.126:35563)
0925 15:23:31.244080 (+ 2119us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:31.244086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:31.244108 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:23:31.245487 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:31.245493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:31.245587 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:31.245703 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:31.247384 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.248239 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.249816 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.250184 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.250324 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:31.253856 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:31.253865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:31.253873 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:31.253916 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:31.257203 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:31.258062 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:31.258070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:31.258076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:31.258219 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:31.258612 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:31.258619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:31.258626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:31.258990 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:31.259142 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:31.259758 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 15:23:31.260001 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1873,"threads_started":1}
I20260925 15:23:31.261040 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:31.242415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35563 (local address 127.8.226.65:33649)
0925 15:23:31.243205 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:31.243252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:31.245218 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:31.245968 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:31.245980 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:31.246536 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:31.247223 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:31.247242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.248391 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.248397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:31.249664 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:31.249675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.250409 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.250418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:31.250641 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:31.251303 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:31.251328 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:31.253634 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:31.257338 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:31.257347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:31.257366 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:31.257938 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:31.258335 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:31.258338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:31.258340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:31.258504 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:31.259105 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:31.259111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:31.259395 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 15:23:31.259612 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 15:23:31.259953 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":337,"threads_started":1}
I20260925 15:23:31.262512 26460 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35563
I20260925 15:23:31.262810 26460 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:31.263489 26460 heartbeater.cc:511] Master 127.8.226.126:35563 requested a full tablet report, sending...
I20260925 15:23:31.265445 26271 ts_manager.cc:194] Registered new tserver with Master: 62f481f4369b4e7fafccbfc2be41e367 (127.8.226.65:34053)
I20260925 15:23:31.267026 26271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:33649
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:23:31.300446 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35563
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:31.618065 26466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:31.618351 26466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:31.618515 26466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:31.625717 26466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:31.625856 26466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:31.626024 26466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:31.635394 26466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:31.636531 26466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:31.637851 26466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:31.650686 26471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:31.650629 26474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:31.650661 26472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:31.651347 26466 server_base.cc:1034] running on GCE node
I20260925 15:23:31.651914 26466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:31.653218 26466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:31.654417 26466 hybrid_clock.cc:648] HybridClock initialized: now 1790349811654364 us; error 46 us; skew 500 ppm
I20260925 15:23:31.654748 26466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:31 dist-test-slave-8vw7 krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:31.657838 26466 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:23:31.659227 26466 webserver.cc:460] Webserver started at http://127.8.226.66:35671/ using document root <none> and password file <none>
I20260925 15:23:31.659861 26466 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:31.659957 26466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:31.660214 26466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:31.663169 26466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "bee93c1187584d179df863eb984a0ff9"
format_stamp: "Formatted at 2026-09-25 15:23:31 on dist-test-slave-8vw7"
server_key: "a24c3c776629eb3b483e7686b10a069b"
server_key_iv: "80f3b16237d575a4d0ad99d0249c01b8"
server_key_version: "encryptionkey@0"
I20260925 15:23:31.663877 26466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "bee93c1187584d179df863eb984a0ff9"
format_stamp: "Formatted at 2026-09-25 15:23:31 on dist-test-slave-8vw7"
server_key: "a24c3c776629eb3b483e7686b10a069b"
server_key_iv: "80f3b16237d575a4d0ad99d0249c01b8"
server_key_version: "encryptionkey@0"
I20260925 15:23:31.668939 26466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 15:23:31.672647 26481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:31.674057 26466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:31.674240 26466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "bee93c1187584d179df863eb984a0ff9"
format_stamp: "Formatted at 2026-09-25 15:23:31 on dist-test-slave-8vw7"
server_key: "a24c3c776629eb3b483e7686b10a069b"
server_key_iv: "80f3b16237d575a4d0ad99d0249c01b8"
server_key_version: "encryptionkey@0"
I20260925 15:23:31.674449 26466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:31.698748 26466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:31.702713 26466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:31.703001 26466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:31.703943 26466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:31.705339 26466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:31.705408 26466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:31.705564 26466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:31.705612 26466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:31.738538 26466 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:36783
I20260925 15:23:31.738590 26593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:36783 every 8 connection(s)
I20260925 15:23:31.739828 26466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:31.742201 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26466
I20260925 15:23:31.742424 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:31.742946 9097 external_mini_cluster.cc:1428] Setting key 8866165d4c03c11162145cac9b202cb1
Sep 25 15:23:31 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:31.760989 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:31.743764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:37389 (local address 127.8.226.126:35563)
0925 15:23:31.743961 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:31.743967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:31.744229 (+ 262us) server_negotiation.cc:407] Connection header received
0925 15:23:31.747259 (+ 3030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:31.747265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:31.747355 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:31.747467 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:31.748969 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.749710 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.751423 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.751863 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.752000 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:31.755508 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:31.755513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:31.755516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:31.755548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:31.758009 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:31.758712 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:31.758719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:31.758722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:31.758850 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:31.759262 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:31.759269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:31.759273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:31.759535 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:31.759692 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:31.760339 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 15:23:31.760600 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:23:31.761956 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:31.743143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35563 (local address 127.8.226.66:37389)
0925 15:23:31.744055 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:31.744102 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:31.746980 (+ 2878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:31.747713 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:31.747721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:31.748221 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:31.748779 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:31.748795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.749862 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.749868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:31.751280 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:31.751296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:31.752083 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:31.752090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:31.752316 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:31.753055 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:31.753076 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:31.755279 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:31.758144 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:31.758153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:31.758169 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:31.758574 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:31.758960 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:31.758965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:31.758968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:31.759147 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:31.759675 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:31.759682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:31.759994 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 15:23:31.760255 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:23:31.760801 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":299,"threads_started":1}
I20260925 15:23:31.763340 26594 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35563
I20260925 15:23:31.763710 26594 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:31.764642 26594 heartbeater.cc:511] Master 127.8.226.126:35563 requested a full tablet report, sending...
I20260925 15:23:31.766328 26271 ts_manager.cc:194] Registered new tserver with Master: bee93c1187584d179df863eb984a0ff9 (127.8.226.66:36783)
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
I20260925 15:23:31.767282 26271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:37389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:23:31.797487 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35563
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:32.116943 26599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:32.117238 26599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:32.117419 26599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:32.124343 26599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:32.124444 26599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:32.124647 26599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:32.134006 26599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:32.135229 26599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:32.136507 26599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:32.148787 26604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:32.148916 26605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:32.149181 26607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:32.150002 26599 server_base.cc:1034] running on GCE node
I20260925 15:23:32.150606 26599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:32.151638 26599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:32.152830 26599 hybrid_clock.cc:648] HybridClock initialized: now 1790349812152794 us; error 26 us; skew 500 ppm
I20260925 15:23:32.153152 26599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:32 dist-test-slave-8vw7 krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:32.156143 26599 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:23:32.157510 26599 webserver.cc:460] Webserver started at http://127.8.226.67:35599/ using document root <none> and password file <none>
I20260925 15:23:32.158087 26599 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:32.158232 26599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:32.158528 26599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:32.161288 26599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "771e764823cc472e8a94ebc9be50be0f"
format_stamp: "Formatted at 2026-09-25 15:23:32 on dist-test-slave-8vw7"
server_key: "4a4982d33a8bb855889f6056cfa6f257"
server_key_iv: "0799fa6f5f163287d10bd4e4b8b36cc6"
server_key_version: "encryptionkey@0"
I20260925 15:23:32.161980 26599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "771e764823cc472e8a94ebc9be50be0f"
format_stamp: "Formatted at 2026-09-25 15:23:32 on dist-test-slave-8vw7"
server_key: "4a4982d33a8bb855889f6056cfa6f257"
server_key_iv: "0799fa6f5f163287d10bd4e4b8b36cc6"
server_key_version: "encryptionkey@0"
I20260925 15:23:32.166841 26599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:23:32.170148 26614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:32.171368 26599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:32.171485 26599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "771e764823cc472e8a94ebc9be50be0f"
format_stamp: "Formatted at 2026-09-25 15:23:32 on dist-test-slave-8vw7"
server_key: "4a4982d33a8bb855889f6056cfa6f257"
server_key_iv: "0799fa6f5f163287d10bd4e4b8b36cc6"
server_key_version: "encryptionkey@0"
I20260925 15:23:32.171634 26599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:32.196455 26599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:32.200412 26599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:32.200714 26599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:32.201690 26599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:32.203006 26599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:32.203110 26599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:32.203264 26599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:32.203337 26599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:32.236305 26599 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:43947
I20260925 15:23:32.236395 26726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:43947 every 8 connection(s)
I20260925 15:23:32.237721 26599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:32.239841 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26599
I20260925 15:23:32.240052 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:32.240633 9097 external_mini_cluster.cc:1428] Setting key 6063a8f910a1927fa2b54a7ce58cd87d
Sep 25 15:23:32 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:32.257696 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.240551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:49885 (local address 127.8.226.126:35563)
0925 15:23:32.240753 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.240759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.242146 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 15:23:32.244324 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.244328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.244393 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.244472 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:32.245909 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.246651 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.248061 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.248444 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.248532 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.251713 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:32.251717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:32.251720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:32.251750 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:32.254567 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.255477 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.255494 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.255497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.255721 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.256172 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.256177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.256180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.256490 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:32.256600 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.257288 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.257443 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:23:32.258565 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.241182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35563 (local address 127.8.226.67:49885)
0925 15:23:32.241986 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:32.242037 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:32.244095 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:32.244664 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:32.244671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:32.245240 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:32.245768 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.245784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.246779 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.246783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.247947 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.247957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.248611 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.248616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.248809 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.249440 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:32.249460 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:32.251528 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:32.254698 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:32.254722 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:32.254737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:32.255335 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:32.255852 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:32.255856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:32.255858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:32.256021 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:32.256607 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:32.256613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:32.256897 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 15:23:32.257183 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 15:23:32.257526 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":466,"threads_started":1}
I20260925 15:23:32.259773 26727 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35563
I20260925 15:23:32.260056 26727 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:32.260684 26727 heartbeater.cc:511] Master 127.8.226.126:35563 requested a full tablet report, sending...
I20260925 15:23:32.262092 26271 ts_manager.cc:194] Registered new tserver with Master: 771e764823cc472e8a94ebc9be50be0f (127.8.226.67:43947)
I20260925 15:23:32.262874 26271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:49885
I20260925 15:23:32.268404 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:32.269784 26460 heartbeater.cc:503] Master 127.8.226.126:35563 was elected leader, sending a full tablet report...
I20260925 15:23:32.285971 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.273159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.8.226.126:35563)
0925 15:23:32.273320 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.273324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.273781 (+ 457us) server_negotiation.cc:407] Connection header received
0925 15:23:32.274000 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.274009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.274065 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.274165 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:32.275708 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.276426 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.277807 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.278242 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.278372 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.280829 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:32.280834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:32.280837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:32.280865 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:32.283405 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.284066 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.284073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.284081 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.284193 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.284558 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.284563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.284566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.284857 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:32.285145 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.285453 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.285662 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:23:32.289584 26271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:32.296130 26271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:32.316672 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.307448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.8.226.65:34053)
0925 15:23:32.308076 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.308084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.308648 (+ 564us) server_negotiation.cc:407] Connection header received
0925 15:23:32.309064 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.309075 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.309330 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.309510 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:32.310467 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.311734 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.313652 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.315175 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.315291 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.315516 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.316139 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.316327 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":301,"threads_started":1}
I20260925 15:23:32.316651 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.307377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:34053 (local address 127.0.0.1:40622)
0925 15:23:32.308506 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:32.308534 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:32.308749 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:32.309857 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:32.309866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:32.309893 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:32.310275 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.310287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.311878 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.311882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.313482 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.313496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.315687 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.315693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.315897 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.315940 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:23:32.316069 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:23:32.316249 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":861,"threads_started":1}
I20260925 15:23:32.320994 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.308801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36783 (local address 127.0.0.1:41934)
0925 15:23:32.309293 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:32.309309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:32.309416 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:32.314383 (+ 4967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:32.314387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:32.314416 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:32.314829 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.314841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.316336 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.316341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.318851 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.318866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.320180 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.320186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.320347 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.320379 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:23:32.320482 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:23:32.320611 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260925 15:23:32.321262 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.309104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.8.226.66:36783)
0925 15:23:32.313811 (+ 4707us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.313819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.313845 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:23:32.313937 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.313944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.314173 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.314986 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:32.315170 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.316184 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.319024 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.319956 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.320120 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.320315 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.320754 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.320922 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4537,"thread_start_us":4433,"threads_started":1}
I20260925 15:23:32.321609 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.310089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.8.226.67:43947)
0925 15:23:32.313454 (+ 3365us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.313463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.313495 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:23:32.313587 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.313597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.313863 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.314761 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:32.315043 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.316588 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.318880 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.320268 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.320439 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.320648 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.321095 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.321265 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3193,"thread_start_us":3050,"threads_started":1}
I20260925 15:23:32.321674 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.309414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:43947 (local address 127.0.0.1:44440)
0925 15:23:32.309956 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:32.309973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:32.310079 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:32.314065 (+ 3986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:32.314069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:32.314099 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:32.314539 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.314552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.316744 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.316749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.318693 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.318707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.320495 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.320501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.320648 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.320685 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:23:32.320776 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:23:32.320888 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":1,"thread_start_us":1001,"threads_started":1}
I20260925 15:23:32.321188 26395 tablet_service.cc:1467] Processing CreateTablet for tablet 432783d1a1de4d649616960acd670f7d (DEFAULT_TABLE table=test-table [id=7067a36a2378491399a9c4501f815202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:32.322779 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432783d1a1de4d649616960acd670f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:32.325150 26529 tablet_service.cc:1467] Processing CreateTablet for tablet 432783d1a1de4d649616960acd670f7d (DEFAULT_TABLE table=test-table [id=7067a36a2378491399a9c4501f815202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:32.325434 26662 tablet_service.cc:1467] Processing CreateTablet for tablet 432783d1a1de4d649616960acd670f7d (DEFAULT_TABLE table=test-table [id=7067a36a2378491399a9c4501f815202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:32.326345 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432783d1a1de4d649616960acd670f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:32.326761 26662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432783d1a1de4d649616960acd670f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:32.338191 26745 tablet_bootstrap.cc:492] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Bootstrap starting.
I20260925 15:23:32.340211 26746 tablet_bootstrap.cc:492] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Bootstrap starting.
I20260925 15:23:32.341682 26747 tablet_bootstrap.cc:492] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Bootstrap starting.
I20260925 15:23:32.341974 26745 tablet_bootstrap.cc:654] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:32.343338 26745 log.cc:824] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:32.343722 26746 tablet_bootstrap.cc:654] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:32.344774 26747 tablet_bootstrap.cc:654] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:32.345221 26746 log.cc:824] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:32.346019 26747 log.cc:824] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:32.348927 26747 tablet_bootstrap.cc:492] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: No bootstrap required, opened a new log
I20260925 15:23:32.349239 26747 ts_tablet_manager.cc:1397] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260925 15:23:32.349268 26745 tablet_bootstrap.cc:492] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: No bootstrap required, opened a new log
I20260925 15:23:32.349375 26746 tablet_bootstrap.cc:492] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: No bootstrap required, opened a new log
I20260925 15:23:32.349500 26745 ts_tablet_manager.cc:1397] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260925 15:23:32.349644 26746 ts_tablet_manager.cc:1397] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260925 15:23:32.357239 26747 raft_consensus.cc:348] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.357589 26747 raft_consensus.cc:374] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:32.357695 26747 raft_consensus.cc:729] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771e764823cc472e8a94ebc9be50be0f, State: Initialized, Role: FOLLOWER
I20260925 15:23:32.358306 26747 consensus_queue.cc:260] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.359346 26727 heartbeater.cc:503] Master 127.8.226.126:35563 was elected leader, sending a full tablet report...
I20260925 15:23:32.360006 26747 ts_tablet_manager.cc:1428] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260925 15:23:32.359979 26746 raft_consensus.cc:348] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.359979 26745 raft_consensus.cc:348] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.360379 26745 raft_consensus.cc:374] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:32.360384 26746 raft_consensus.cc:374] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:32.360468 26746 raft_consensus.cc:729] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f481f4369b4e7fafccbfc2be41e367, State: Initialized, Role: FOLLOWER
I20260925 15:23:32.360467 26745 raft_consensus.cc:729] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee93c1187584d179df863eb984a0ff9, State: Initialized, Role: FOLLOWER
I20260925 15:23:32.361063 26746 consensus_queue.cc:260] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.361063 26745 consensus_queue.cc:260] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.362735 26745 ts_tablet_manager.cc:1428] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:23:32.363050 26746 ts_tablet_manager.cc:1428] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:23:32.363543 26594 heartbeater.cc:503] Master 127.8.226.126:35563 was elected leader, sending a full tablet report...
I20260925 15:23:32.405658 26752 raft_consensus.cc:482] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:32.405984 26752 raft_consensus.cc:504] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.407785 26752 leader_election.cc:288] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bee93c1187584d179df863eb984a0ff9 (127.8.226.66:36783), 771e764823cc472e8a94ebc9be50be0f (127.8.226.67:43947)
I20260925 15:23:32.414630 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.408076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:40495 (local address 127.8.226.66:36783)
0925 15:23:32.408273 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.408278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.408765 (+ 487us) server_negotiation.cc:407] Connection header received
0925 15:23:32.409005 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.409013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.409103 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.409203 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:32.409843 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.410774 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.412745 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.413668 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.413795 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.413845 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.414250 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.414376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:23:32.414739 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.408088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:36783 (local address 127.8.226.65:40495)
0925 15:23:32.408657 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:32.408672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:32.408839 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:32.409248 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:32.409255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:32.409289 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:32.409675 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.409687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.410908 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.410913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.412584 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.412599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.413882 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.413890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.414071 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.414108 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:23:32.414214 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:23:32.414331 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":36,"thread_start_us":320,"threads_started":1}
I20260925 15:23:32.414940 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.409033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:53457 (local address 127.8.226.67:43947)
0925 15:23:32.409230 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.409236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.409522 (+ 286us) server_negotiation.cc:407] Connection header received
0925 15:23:32.409642 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.409646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.409735 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.409823 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:32.410406 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.411329 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.413123 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.413978 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.414123 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.414210 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.414535 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.414673 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:23:32.415226 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.408785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:43947 (local address 127.8.226.65:53457)
0925 15:23:32.409418 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:32.409438 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:32.409530 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:32.409846 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:32.409850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:32.409872 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:32.410259 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.410269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.411456 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.411459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.412966 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:32.412998 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.414168 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.414172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:32.414364 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.414394 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:23:32.414487 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:23:32.414602 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":6,"thread_start_us":451,"threads_started":1}
I20260925 15:23:32.415792 26549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432783d1a1de4d649616960acd670f7d" candidate_uuid: "62f481f4369b4e7fafccbfc2be41e367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bee93c1187584d179df863eb984a0ff9" is_pre_election: true
I20260925 15:23:32.416102 26682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432783d1a1de4d649616960acd670f7d" candidate_uuid: "62f481f4369b4e7fafccbfc2be41e367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771e764823cc472e8a94ebc9be50be0f" is_pre_election: true
I20260925 15:23:32.416255 26549 raft_consensus.cc:2454] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f481f4369b4e7fafccbfc2be41e367 in term 0.
I20260925 15:23:32.416585 26682 raft_consensus.cc:2454] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f481f4369b4e7fafccbfc2be41e367 in term 0.
I20260925 15:23:32.416910 26351 leader_election.cc:302] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f481f4369b4e7fafccbfc2be41e367, bee93c1187584d179df863eb984a0ff9; no voters:
I20260925 15:23:32.417294 26752 raft_consensus.cc:2781] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:32.417467 26752 raft_consensus.cc:482] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:32.417583 26752 raft_consensus.cc:3037] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:32.419723 26752 raft_consensus.cc:504] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.420275 26752 leader_election.cc:288] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [CANDIDATE]: Term 1 election: Requested vote from peers bee93c1187584d179df863eb984a0ff9 (127.8.226.66:36783), 771e764823cc472e8a94ebc9be50be0f (127.8.226.67:43947)
I20260925 15:23:32.420679 26549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432783d1a1de4d649616960acd670f7d" candidate_uuid: "62f481f4369b4e7fafccbfc2be41e367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bee93c1187584d179df863eb984a0ff9"
I20260925 15:23:32.420781 26682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432783d1a1de4d649616960acd670f7d" candidate_uuid: "62f481f4369b4e7fafccbfc2be41e367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771e764823cc472e8a94ebc9be50be0f"
I20260925 15:23:32.420884 26549 raft_consensus.cc:3037] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:32.421036 26682 raft_consensus.cc:3037] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:32.423099 26549 raft_consensus.cc:2454] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f481f4369b4e7fafccbfc2be41e367 in term 1.
I20260925 15:23:32.423197 26682 raft_consensus.cc:2454] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f481f4369b4e7fafccbfc2be41e367 in term 1.
I20260925 15:23:32.423471 26351 leader_election.cc:302] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f481f4369b4e7fafccbfc2be41e367, bee93c1187584d179df863eb984a0ff9; no voters:
I20260925 15:23:32.423739 26752 raft_consensus.cc:2781] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:32.424168 26752 raft_consensus.cc:686] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 1 LEADER]: Becoming Leader. State: Replica: 62f481f4369b4e7fafccbfc2be41e367, State: Running, Role: LEADER
I20260925 15:23:32.424638 26752 consensus_queue.cc:237] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } }
I20260925 15:23:32.429646 26270 catalog_manager.cc:5495] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f481f4369b4e7fafccbfc2be41e367 (127.8.226.65). New cstate: current_term: 1 leader_uuid: "62f481f4369b4e7fafccbfc2be41e367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f481f4369b4e7fafccbfc2be41e367" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 } health_report { overall_health: UNKNOWN } } }
I20260925 15:23:32.471249 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.465822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.8.226.65:34053)
0925 15:23:32.466124 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.466130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.466148 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:23:32.466294 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.466298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.466390 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.466550 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:32.466924 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.467766 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.469125 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.469523 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.469633 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.470224 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:32.470770 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:32.470873 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.470946 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.471006 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260925 15:23:32.479395 26549 raft_consensus.cc:1261] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 1 FOLLOWER]: Refusing update from remote peer 62f481f4369b4e7fafccbfc2be41e367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:32.479394 26682 raft_consensus.cc:1261] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 1 FOLLOWER]: Refusing update from remote peer 62f481f4369b4e7fafccbfc2be41e367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:32.480329 26756 consensus_queue.cc:1035] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771e764823cc472e8a94ebc9be50be0f" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 43947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:32.480788 26752 consensus_queue.cc:1035] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bee93c1187584d179df863eb984a0ff9" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:23:32.489367 26728 tablet.cc:2369] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:23:32.491817 26595 tablet.cc:2369] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:23:32.492708 26461 tablet.cc:2369] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:32.493687 26759 mvcc.cc:205] Tried to move back new op lower bound from 7333272831905312768 to 7333272831695409152. Current Snapshot: MvccSnapshot[applied={T|T < 7333272831905312768}]
I20260925 15:23:32.507999 26271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:32.508309 26271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:32.511742 26271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b09f1def95d0481fb7de0a392b878778: Sending DeleteTablet for 3 replicas of tablet 432783d1a1de4d649616960acd670f7d
I20260925 15:23:32.512775 26395 tablet_service.cc:1514] Processing DeleteTablet for tablet 432783d1a1de4d649616960acd670f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:32 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:40622
I20260925 15:23:32.513019 26662 tablet_service.cc:1514] Processing DeleteTablet for tablet 432783d1a1de4d649616960acd670f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:32 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:44440
I20260925 15:23:32.513443 26529 tablet_service.cc:1514] Processing DeleteTablet for tablet 432783d1a1de4d649616960acd670f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:32 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:41934
I20260925 15:23:32.513903 26771 tablet_replica.cc:330] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: stopping tablet replica
I20260925 15:23:32.514389 26771 raft_consensus.cc:2229] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:32.514629 26770 tablet_replica.cc:330] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: stopping tablet replica
I20260925 15:23:32.515031 26771 raft_consensus.cc:2258] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:32.515064 26770 raft_consensus.cc:2229] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:32.515339 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26332
I20260925 15:23:32.515705 26770 raft_consensus.cc:2258] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:32.517972 26771 ts_tablet_manager.cc:1905] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:32.518515 26770 ts_tablet_manager.cc:1905] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:32.526000 26772 tablet_replica.cc:330] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: stopping tablet replica
I20260925 15:23:32.526535 26772 raft_consensus.cc:2229] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:32.526787 26771 ts_tablet_manager.cc:1918] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:32.526981 26771 log.cc:1196] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/432783d1a1de4d649616960acd670f7d
I20260925 15:23:32.527112 26772 raft_consensus.cc:2258] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:32.527551 26771 ts_tablet_manager.cc:1939] T 432783d1a1de4d649616960acd670f7d P 771e764823cc472e8a94ebc9be50be0f: Deleting consensus metadata
I20260925 15:23:32.529805 26258 catalog_manager.cc:4841] TS 771e764823cc472e8a94ebc9be50be0f (127.8.226.67:43947): tablet 432783d1a1de4d649616960acd670f7d (table test-table [id=7067a36a2378491399a9c4501f815202]) successfully deleted
I20260925 15:23:32.530548 26770 ts_tablet_manager.cc:1918] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:32.530717 26770 log.cc:1196] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/432783d1a1de4d649616960acd670f7d
I20260925 15:23:32.530838 26772 ts_tablet_manager.cc:1905] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:32.531252 26770 ts_tablet_manager.cc:1939] T 432783d1a1de4d649616960acd670f7d P 62f481f4369b4e7fafccbfc2be41e367: Deleting consensus metadata
I20260925 15:23:32.532565 26258 catalog_manager.cc:4841] TS 62f481f4369b4e7fafccbfc2be41e367 (127.8.226.65:34053): tablet 432783d1a1de4d649616960acd670f7d (table test-table [id=7067a36a2378491399a9c4501f815202]) successfully deleted
Sep 25 15:23:32 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:23:32.537580 26772 ts_tablet_manager.cc:1918] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:32.537756 26772 log.cc:1196] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/432783d1a1de4d649616960acd670f7d
I20260925 15:23:32.538264 26772 ts_tablet_manager.cc:1939] T 432783d1a1de4d649616960acd670f7d P bee93c1187584d179df863eb984a0ff9: Deleting consensus metadata
I20260925 15:23:32.539979 26258 catalog_manager.cc:4841] TS bee93c1187584d179df863eb984a0ff9 (127.8.226.66:36783): tablet 432783d1a1de4d649616960acd670f7d (table test-table [id=7067a36a2378491399a9c4501f815202]) successfully deleted
I20260925 15:23:32.543159 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.516356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.8.226.65:34053)
0925 15:23:32.516529 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.516535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.517016 (+ 481us) server_negotiation.cc:407] Connection header received
0925 15:23:32.518901 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.518912 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.518993 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.519065 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:32.525063 (+ 5998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.526287 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.528516 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.529310 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.529424 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.536912 (+ 7488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:32.536922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:32.536935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:32.537026 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:32.540362 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.541035 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.541041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.541044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.541197 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.541600 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.541609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.541611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.541970 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:32.542311 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.542579 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.542824 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:23:32.543980 26455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:32.646828 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26332
I20260925 15:23:32.665704 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26466
Sep 25 15:23:32 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:23:32.681874 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.666510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.8.226.66:36783)
0925 15:23:32.666674 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.666680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.666698 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:23:32.666865 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.666869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.666934 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.667084 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:32.668354 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.669202 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.670535 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.670931 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.671035 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.675466 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:32.675481 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:32.675517 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:32.675588 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:32.679197 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.679998 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.680005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.680008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.680143 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.680475 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.680478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.680480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.680913 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:32.681189 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.681458 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.681619 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:23:32.682526 26589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:32.771895 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26466
I20260925 15:23:32.788784 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26599
Sep 25 15:23:32 dist-test-slave-8vw7 krb5kdc[26226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:23:32.803315 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:32.789609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.8.226.67:43947)
0925 15:23:32.789793 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:32.789798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:32.789822 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:23:32.789960 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:32.789965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:32.790040 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:32.790121 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:32.791406 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.792107 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.793438 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:32.793816 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:32.793906 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:32.798213 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:32.798225 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:32.798252 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:32.798281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:32.800765 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.801492 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.801499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.801501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.801637 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:32.801938 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:32.801941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:32.801943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:32.802287 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:32.802502 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:32.802801 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 15:23:32.802971 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:23:32.803998 26722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:32.893106 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26599
I20260925 15:23:32.909297 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26239
I20260925 15:23:32.910458 26301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:33.024185 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26239
2026-09-25T15:23:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:23:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:33Z Disabled control of system clock
I20260925 15:23:33.105149 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38759
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:23:33.427192 26781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:33.427481 26781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:33.427558 26781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:33.434716 26781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:33.434819 26781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:33.434856 26781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:33.434912 26781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:33.434955 26781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:33.443746 26781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:38759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:33.445022 26781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:33.446656 26781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:33.458158 26786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:33.458233 26789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:33.458259 26787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:33.458906 26781 server_base.cc:1034] running on GCE node
I20260925 15:23:33.459646 26781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:33.460906 26781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:33.462146 26781 hybrid_clock.cc:648] HybridClock initialized: now 1790349813462116 us; error 31 us; skew 500 ppm
I20260925 15:23:33.462493 26781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:33.464743 26781 webserver.cc:460] Webserver started at http://127.8.226.126:41581/ using document root <none> and password file <none>
I20260925 15:23:33.465452 26781 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:33.465569 26781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:33.465992 26781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:33.469842 26781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "1f13d631587545ddb303e9f76ffbc3ae"
format_stamp: "Formatted at 2026-09-25 15:23:33 on dist-test-slave-8vw7"
server_key: "ccfe1bd16105f350f6c38c4068e7b637"
server_key_iv: "02da7b4c4a5d44fb44a64c4a53c15943"
server_key_version: "encryptionkey@0"
I20260925 15:23:33.470558 26781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "1f13d631587545ddb303e9f76ffbc3ae"
format_stamp: "Formatted at 2026-09-25 15:23:33 on dist-test-slave-8vw7"
server_key: "ccfe1bd16105f350f6c38c4068e7b637"
server_key_iv: "02da7b4c4a5d44fb44a64c4a53c15943"
server_key_version: "encryptionkey@0"
I20260925 15:23:33.475359 26781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:23:33.478498 26795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:33.479794 26781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:23:33.479970 26781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "1f13d631587545ddb303e9f76ffbc3ae"
format_stamp: "Formatted at 2026-09-25 15:23:33 on dist-test-slave-8vw7"
server_key: "ccfe1bd16105f350f6c38c4068e7b637"
server_key_iv: "02da7b4c4a5d44fb44a64c4a53c15943"
server_key_version: "encryptionkey@0"
I20260925 15:23:33.480187 26781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:33.497303 26781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:33.498054 26781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:33.498329 26781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:33.516341 26781 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:38759
I20260925 15:23:33.516345 26846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:38759 every 8 connection(s)
I20260925 15:23:33.517689 26781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:33.521286 26847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:33.525645 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26781
I20260925 15:23:33.525849 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:33.526324 9097 external_mini_cluster.cc:1428] Setting key e6d431fb4b2fd97adce9a66a42cd9c1d
I20260925 15:23:33.533330 26847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae: Bootstrap starting.
I20260925 15:23:33.536914 26847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:33.538280 26847 log.cc:824] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:33.540918 26847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae: No bootstrap required, opened a new log
I20260925 15:23:33.548501 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:33.528181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.8.226.126:38759)
0925 15:23:33.529322 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:33.529336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:33.529390 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:23:33.531854 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:33.531880 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:33.532456 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:33.532881 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:33.533450 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:33.535446 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:33.536817 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:33.537903 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:33.538105 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:33.545673 (+ 7568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:33.545683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:33.545702 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:33.546423 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:33.546531 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:33.546563 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:33.546693 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:33.546893 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:23:33.547235 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":377,"threads_started":1}
I20260925 15:23:33.549172 26847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f13d631587545ddb303e9f76ffbc3ae" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38759 } }
I20260925 15:23:33.549531 26847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:33.549660 26847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f13d631587545ddb303e9f76ffbc3ae, State: Initialized, Role: FOLLOWER
I20260925 15:23:33.550338 26847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f13d631587545ddb303e9f76ffbc3ae" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38759 } }
I20260925 15:23:33.550616 26847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:33.550766 26847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:33.550947 26847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:33.553566 26847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f13d631587545ddb303e9f76ffbc3ae" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38759 } }
I20260925 15:23:33.554114 26847 leader_election.cc:302] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f13d631587545ddb303e9f76ffbc3ae; no voters:
I20260925 15:23:33.554615 26847 leader_election.cc:288] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:33.554761 26852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:33.555518 26852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [term 1 LEADER]: Becoming Leader. State: Replica: 1f13d631587545ddb303e9f76ffbc3ae, State: Running, Role: LEADER
I20260925 15:23:33.556039 26852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f13d631587545ddb303e9f76ffbc3ae" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38759 } }
I20260925 15:23:33.556723 26847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:33.558970 26853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f13d631587545ddb303e9f76ffbc3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f13d631587545ddb303e9f76ffbc3ae" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38759 } } }
I20260925 15:23:33.558945 26854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f13d631587545ddb303e9f76ffbc3ae. Latest consensus state: current_term: 1 leader_uuid: "1f13d631587545ddb303e9f76ffbc3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f13d631587545ddb303e9f76ffbc3ae" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 38759 } } }
I20260925 15:23:33.559407 26853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:33.559407 26854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:33.561390 26857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:33.566282 26857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:33.575629 26857 catalog_manager.cc:1347] Generated new cluster ID: 06e295ab64734f52a7e32dc882e1f1b9
I20260925 15:23:33.575733 26857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:33.586962 26857 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:33.588721 26857 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:33.603354 26857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae: Generated new TSK 0
I20260925 15:23:33.604233 26857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:33.614405 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38759
--builtin_ntp_servers=127.8.226.84:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:23:33.947654 26871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:33.947968 26871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:33.948113 26871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:33.955587 26871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:33.955696 26871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:33.955873 26871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:33.965884 26871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:33.967203 26871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:33.969079 26871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:33.982962 26877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:33.983168 26876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:33.983048 26879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:33.984133 26871 server_base.cc:1034] running on GCE node
I20260925 15:23:33.984834 26871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:33.985800 26871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:33.987004 26871 hybrid_clock.cc:648] HybridClock initialized: now 1790349813986953 us; error 47 us; skew 500 ppm
I20260925 15:23:33.987329 26871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:33.989703 26871 webserver.cc:460] Webserver started at http://127.8.226.65:43281/ using document root <none> and password file <none>
I20260925 15:23:33.990406 26871 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:33.990522 26871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:33.990864 26871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:33.993806 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "560e0de098f0400785b82e6f1f15b1eb"
format_stamp: "Formatted at 2026-09-25 15:23:33 on dist-test-slave-8vw7"
server_key: "1015322a0244f954edc4c7bcd9ad9e04"
server_key_iv: "5985e6229541de77ba2bd7fb838ac3d9"
server_key_version: "encryptionkey@0"
I20260925 15:23:33.994596 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "560e0de098f0400785b82e6f1f15b1eb"
format_stamp: "Formatted at 2026-09-25 15:23:33 on dist-test-slave-8vw7"
server_key: "1015322a0244f954edc4c7bcd9ad9e04"
server_key_iv: "5985e6229541de77ba2bd7fb838ac3d9"
server_key_version: "encryptionkey@0"
I20260925 15:23:33.999667 26871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:23:34.003371 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.004755 26871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:23:34.004940 26871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "560e0de098f0400785b82e6f1f15b1eb"
format_stamp: "Formatted at 2026-09-25 15:23:33 on dist-test-slave-8vw7"
server_key: "1015322a0244f954edc4c7bcd9ad9e04"
server_key_iv: "5985e6229541de77ba2bd7fb838ac3d9"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.005229 26871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:34.019763 26871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:34.020545 26871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:34.020859 26871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:34.021873 26871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:34.023319 26871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:34.023406 26871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.023550 26871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:34.023645 26871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.055637 26871 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:43757
I20260925 15:23:34.055716 26997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:43757 every 8 connection(s)
I20260925 15:23:34.057083 26871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:34.061676 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 26871
I20260925 15:23:34.061854 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:34.062326 9097 external_mini_cluster.cc:1428] Setting key 3a3f1800286ed37ec7eeed96f387b42e
I20260925 15:23:34.065039 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38759
--builtin_ntp_servers=127.8.226.84:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:23:34.071262 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.059802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:34267 (local address 127.8.226.126:38759)
0925 15:23:34.060218 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:34.060222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:34.061580 (+ 1358us) server_negotiation.cc:407] Connection header received
0925 15:23:34.063633 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:34.063639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:34.063729 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:34.063852 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:34.065059 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.066266 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.067830 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.068551 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.068694 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.070330 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:34.070339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:34.070347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:34.070376 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:34.070451 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:34.070470 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:34.070558 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:34.070908 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 15:23:34.071001 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
I20260925 15:23:34.072722 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.060425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38759 (local address 127.8.226.65:34267)
0925 15:23:34.061426 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:34.061467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:34.063362 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:34.064154 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:34.064167 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:34.064274 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:34.064851 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:34.064869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.066422 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.066426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:34.067674 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:34.067686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.068794 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.068800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:34.069125 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.069931 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:34.069934 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:34.069976 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:34.070059 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:34.070613 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:34.070628 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:34.070639 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:23:34.070830 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:23:34.071288 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":269,"threads_started":1}
I20260925 15:23:34.074970 26998 heartbeater.cc:347] Connected to a master server at 127.8.226.126:38759
I20260925 15:23:34.075296 26998 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:34.076045 26998 heartbeater.cc:511] Master 127.8.226.126:38759 requested a full tablet report, sending...
I20260925 15:23:34.078104 26812 ts_manager.cc:194] Registered new tserver with Master: 560e0de098f0400785b82e6f1f15b1eb (127.8.226.65:43757)
I20260925 15:23:34.079579 26812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.65:34267
W20260925 15:23:34.392728 27002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:34.393077 27002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:34.393236 27002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:34.400441 27002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:34.400560 27002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:34.400735 27002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:34.409822 27002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:34.411161 27002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:34.412881 27002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:34.425606 27007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:34.425575 27010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:34.425657 27008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:34.426172 27002 server_base.cc:1034] running on GCE node
I20260925 15:23:34.426793 27002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:34.427696 27002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:34.428916 27002 hybrid_clock.cc:648] HybridClock initialized: now 1790349814428873 us; error 38 us; skew 500 ppm
I20260925 15:23:34.429239 27002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:34.431408 27002 webserver.cc:460] Webserver started at http://127.8.226.66:40319/ using document root <none> and password file <none>
I20260925 15:23:34.432063 27002 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:34.432184 27002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:34.432493 27002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:34.435266 27002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "c2d76a0abfe749fd880b11465d743921"
format_stamp: "Formatted at 2026-09-25 15:23:34 on dist-test-slave-8vw7"
server_key: "4d804cb4fcff43328cf422dba1025282"
server_key_iv: "4ff9e407c06f20cc1cb9a852d9124800"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.436028 27002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "c2d76a0abfe749fd880b11465d743921"
format_stamp: "Formatted at 2026-09-25 15:23:34 on dist-test-slave-8vw7"
server_key: "4d804cb4fcff43328cf422dba1025282"
server_key_iv: "4ff9e407c06f20cc1cb9a852d9124800"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.440717 27002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:23:34.443948 27016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.445160 27002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:23:34.445315 27002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "c2d76a0abfe749fd880b11465d743921"
format_stamp: "Formatted at 2026-09-25 15:23:34 on dist-test-slave-8vw7"
server_key: "4d804cb4fcff43328cf422dba1025282"
server_key_iv: "4ff9e407c06f20cc1cb9a852d9124800"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.445513 27002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:34.473398 27002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:34.474251 27002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:34.474586 27002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:34.475507 27002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:34.476887 27002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:34.477035 27002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.477159 27002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:34.477243 27002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.509177 27002 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:32959
I20260925 15:23:34.509225 27128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:32959 every 8 connection(s)
I20260925 15:23:34.510483 27002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:34.511631 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27002
I20260925 15:23:34.511811 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:34.512380 9097 external_mini_cluster.cc:1428] Setting key 67aa669ed6d56918a6de08f18b2878a8
I20260925 15:23:34.515296 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38759
--builtin_ntp_servers=127.8.226.84:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:23:34.525566 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.513249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:41413 (local address 127.8.226.126:38759)
0925 15:23:34.513434 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:34.513438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:34.515151 (+ 1713us) server_negotiation.cc:407] Connection header received
0925 15:23:34.517670 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:34.517677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:34.517751 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:34.517832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:34.519360 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.520490 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.522783 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.523203 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.523342 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.524686 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:34.524690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:34.524694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:34.524712 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:34.524741 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:34.524753 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:34.524848 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:34.525116 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 15:23:34.525214 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:23:34.526639 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.513948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38759 (local address 127.8.226.66:41413)
0925 15:23:34.515013 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:34.515056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:34.517371 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:34.518310 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:34.518322 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:34.518428 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:34.519154 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:34.519179 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.520683 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.520688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:34.522639 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:34.522653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.523439 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.523447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:34.523685 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.524372 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:34.524374 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:34.524416 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:34.524487 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:34.524847 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:34.524858 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:34.524868 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:23:34.525101 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:23:34.525485 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":347,"threads_started":1}
I20260925 15:23:34.528055 27129 heartbeater.cc:347] Connected to a master server at 127.8.226.126:38759
I20260925 15:23:34.528368 27129 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:34.529107 27129 heartbeater.cc:511] Master 127.8.226.126:38759 requested a full tablet report, sending...
I20260925 15:23:34.530457 26812 ts_manager.cc:194] Registered new tserver with Master: c2d76a0abfe749fd880b11465d743921 (127.8.226.66:32959)
I20260925 15:23:34.531288 26812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.66:41413
W20260925 15:23:34.844640 27132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:34.844930 27132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:34.845085 27132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:34.852589 27132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:34.852689 27132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:34.852909 27132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:34.862231 27132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:34.863508 27132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:34.865279 27132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:34.878209 27137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:34.878281 27140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:34.878257 27138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:34.879055 27132 server_base.cc:1034] running on GCE node
I20260925 15:23:34.879595 27132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:34.880481 27132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:34.881670 27132 hybrid_clock.cc:648] HybridClock initialized: now 1790349814881670 us; error 84 us; skew 500 ppm
I20260925 15:23:34.881961 27132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:34.884151 27132 webserver.cc:460] Webserver started at http://127.8.226.67:37289/ using document root <none> and password file <none>
I20260925 15:23:34.884857 27132 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:34.884940 27132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:34.885278 27132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:34.887966 27132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "dcba398c4fd542aca092e6013a61e253"
format_stamp: "Formatted at 2026-09-25 15:23:34 on dist-test-slave-8vw7"
server_key: "6cdfb0939c9c2431a57ddbaf6086726b"
server_key_iv: "f24458c7f9cae52417eb01e94c6f9092"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.888674 27132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "dcba398c4fd542aca092e6013a61e253"
format_stamp: "Formatted at 2026-09-25 15:23:34 on dist-test-slave-8vw7"
server_key: "6cdfb0939c9c2431a57ddbaf6086726b"
server_key_iv: "f24458c7f9cae52417eb01e94c6f9092"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.893507 27132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:23:34.896620 27146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.897946 27132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:34.898063 27132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "dcba398c4fd542aca092e6013a61e253"
format_stamp: "Formatted at 2026-09-25 15:23:34 on dist-test-slave-8vw7"
server_key: "6cdfb0939c9c2431a57ddbaf6086726b"
server_key_iv: "f24458c7f9cae52417eb01e94c6f9092"
server_key_version: "encryptionkey@0"
I20260925 15:23:34.898226 27132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:34.918286 27132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:34.919034 27132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:34.919298 27132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:34.920145 27132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:34.921551 27132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:34.921633 27132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.921708 27132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:34.921761 27132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:34.955129 27132 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40729
I20260925 15:23:34.955261 27258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40729 every 8 connection(s)
I20260925 15:23:34.956529 27132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:34.964053 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27132
I20260925 15:23:34.964228 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:34.964689 9097 external_mini_cluster.cc:1428] Setting key 46f59ab9b6b60e1b8f57f1854aac5841
I20260925 15:23:34.970021 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.959215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:47093 (local address 127.8.226.126:38759)
0925 15:23:34.959397 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:34.959402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:34.960820 (+ 1418us) server_negotiation.cc:407] Connection header received
0925 15:23:34.962585 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:34.962590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:34.962659 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:34.962757 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:34.963832 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.964711 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.967030 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.967615 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.967741 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.969140 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:34.969144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:34.969148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:34.969174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:34.969206 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:34.969218 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:34.969292 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:34.969613 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 15:23:34.969794 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:23:34.971076 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.959783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:38759 (local address 127.8.226.67:47093)
0925 15:23:34.960656 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:34.960692 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:34.962314 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:34.962956 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:34.962964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:34.963049 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:34.963634 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:34.963674 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.964895 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.964910 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:34.966867 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:34.966882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.967814 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.967819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:34.968022 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.968767 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:34.968770 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:34.968809 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:34.968888 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:34.969392 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:34.969406 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:34.969413 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:23:34.969580 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:23:34.969934 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":256,"threads_started":1}
I20260925 15:23:34.972368 27259 heartbeater.cc:347] Connected to a master server at 127.8.226.126:38759
I20260925 15:23:34.972715 27259 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:34.973510 27259 heartbeater.cc:511] Master 127.8.226.126:38759 requested a full tablet report, sending...
I20260925 15:23:34.974835 26812 ts_manager.cc:194] Registered new tserver with Master: dcba398c4fd542aca092e6013a61e253 (127.8.226.67:40729)
I20260925 15:23:34.975631 26812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.67:47093
I20260925 15:23:34.979727 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:34.989384 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:34.983784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.8.226.126:38759)
0925 15:23:34.983952 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:34.983957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:34.984418 (+ 461us) server_negotiation.cc:407] Connection header received
0925 15:23:34.984572 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:34.984577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:34.984711 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:34.984794 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:34.985280 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.985966 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.987308 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:34.987695 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:34.987783 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:34.988746 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:34.988751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:34.988754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:34.988771 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:34.988800 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:34.988811 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:34.988892 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:34.989113 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:23:34.989182 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:23:34.993031 26812 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:34.999951 26812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:35.024606 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.012712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.8.226.65:43757)
0925 15:23:35.014556 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.014569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.014605 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:23:35.014732 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.014743 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.015057 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.015314 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:35.017171 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.019155 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.022209 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.023409 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.023548 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.023793 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.024072 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.024254 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":1536,"threads_started":1}
I20260925 15:23:35.024634 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.012593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51316)
0925 15:23:35.013628 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.013678 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:35.013897 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:35.015341 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:35.015352 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:35.015398 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:35.016316 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.016332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.019306 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.019311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.022038 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.022050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.023601 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.023607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.023800 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.023858 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:23:35.024007 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:23:35.024204 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":220,"thread_start_us":3174,"threads_started":1}
I20260925 15:23:35.025897 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.016595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.8.226.66:32959)
0925 15:23:35.017180 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.017187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.017213 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:23:35.017319 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.017328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.017561 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.017773 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:35.018737 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.019894 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.022931 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.024048 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.024162 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.024399 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.025371 (+ 972us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.025540 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":304,"threads_started":1}
I20260925 15:23:35.025943 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.013179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.8.226.67:40729)
0925 15:23:35.016174 (+ 2995us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.016182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.017214 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 15:23:35.017395 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.017402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.017588 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.017792 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:35.018657 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.019992 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.022853 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.024189 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.024320 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.024567 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.025409 (+ 842us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.025588 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2835,"thread_start_us":2686,"threads_started":1}
I20260925 15:23:35.026510 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.016179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:32959 (local address 127.0.0.1:37942)
0925 15:23:35.016851 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.016869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:35.016983 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:35.018064 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:35.018069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:35.018110 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:35.018535 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.018548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.020032 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.020038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.022766 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.022782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.024518 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.024524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.024738 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.024772 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:23:35.024859 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:23:35.025003 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":149,"spinlock_wait_cycles":1920,"thread_start_us":401,"threads_started":1}
I20260925 15:23:35.029217 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.012707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40729 (local address 127.0.0.1:46878)
0925 15:23:35.016621 (+ 3914us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.016647 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:35.016787 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:35.018065 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:35.018068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:35.018097 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:35.018468 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.018480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.020100 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.020104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.022630 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.022646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.024724 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.024730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.024926 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.024987 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:23:35.025078 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:23:35.025212 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3312,"mutex_wait_us":104,"spinlock_wait_cycles":2944,"thread_start_us":678,"threads_started":1}
I20260925 15:23:35.028997 26933 tablet_service.cc:1467] Processing CreateTablet for tablet ea8e19d3de1f45f6b60754641240035a (DEFAULT_TABLE table=test-table [id=d1286be6211a4be99ccf3a4a9881ff51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:35.029527 27064 tablet_service.cc:1467] Processing CreateTablet for tablet ea8e19d3de1f45f6b60754641240035a (DEFAULT_TABLE table=test-table [id=d1286be6211a4be99ccf3a4a9881ff51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:35.030315 26933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8e19d3de1f45f6b60754641240035a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:35.030486 27064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8e19d3de1f45f6b60754641240035a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:35.033946 27194 tablet_service.cc:1467] Processing CreateTablet for tablet ea8e19d3de1f45f6b60754641240035a (DEFAULT_TABLE table=test-table [id=d1286be6211a4be99ccf3a4a9881ff51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:35.035025 27194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8e19d3de1f45f6b60754641240035a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:35.045662 27277 tablet_bootstrap.cc:492] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: Bootstrap starting.
I20260925 15:23:35.050005 27277 tablet_bootstrap.cc:654] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:35.050297 27279 tablet_bootstrap.cc:492] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Bootstrap starting.
I20260925 15:23:35.050729 27278 tablet_bootstrap.cc:492] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Bootstrap starting.
I20260925 15:23:35.051390 27277 log.cc:824] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:35.054113 27279 tablet_bootstrap.cc:654] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:35.054126 27278 tablet_bootstrap.cc:654] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:35.054348 27277 tablet_bootstrap.cc:492] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: No bootstrap required, opened a new log
I20260925 15:23:35.054627 27277 ts_tablet_manager.cc:1397] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:23:35.055414 27278 log.cc:824] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:35.055698 27279 log.cc:824] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:35.061545 27278 tablet_bootstrap.cc:492] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: No bootstrap required, opened a new log
I20260925 15:23:35.061647 27279 tablet_bootstrap.cc:492] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: No bootstrap required, opened a new log
I20260925 15:23:35.061864 27278 ts_tablet_manager.cc:1397] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:23:35.061918 27279 ts_tablet_manager.cc:1397] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260925 15:23:35.065663 27277 raft_consensus.cc:348] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.066145 27277 raft_consensus.cc:374] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:35.066249 27277 raft_consensus.cc:729] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560e0de098f0400785b82e6f1f15b1eb, State: Initialized, Role: FOLLOWER
I20260925 15:23:35.067044 27277 consensus_queue.cc:260] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.072325 27278 raft_consensus.cc:348] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.072844 27278 raft_consensus.cc:374] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:35.072970 27278 raft_consensus.cc:729] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcba398c4fd542aca092e6013a61e253, State: Initialized, Role: FOLLOWER
I20260925 15:23:35.073738 27278 consensus_queue.cc:260] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.073913 27279 raft_consensus.cc:348] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.074468 27279 raft_consensus.cc:374] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:35.074577 27279 raft_consensus.cc:729] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d76a0abfe749fd880b11465d743921, State: Initialized, Role: FOLLOWER
I20260925 15:23:35.075204 27279 consensus_queue.cc:260] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.076292 27277 ts_tablet_manager.cc:1428] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260925 15:23:35.076920 26998 heartbeater.cc:503] Master 127.8.226.126:38759 was elected leader, sending a full tablet report...
I20260925 15:23:35.079931 27259 heartbeater.cc:503] Master 127.8.226.126:38759 was elected leader, sending a full tablet report...
I20260925 15:23:35.080482 27278 ts_tablet_manager.cc:1428] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260925 15:23:35.081766 27129 heartbeater.cc:503] Master 127.8.226.126:38759 was elected leader, sending a full tablet report...
I20260925 15:23:35.082309 27279 ts_tablet_manager.cc:1428] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Time spent starting tablet: real 0.020s user 0.010s sys 0.009s
I20260925 15:23:35.149223 27284 raft_consensus.cc:482] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:35.149530 27284 raft_consensus.cc:504] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.151185 27284 leader_election.cc:288] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 560e0de098f0400785b82e6f1f15b1eb (127.8.226.65:43757), c2d76a0abfe749fd880b11465d743921 (127.8.226.66:32959)
I20260925 15:23:35.157859 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.151690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:47965 (local address 127.8.226.65:43757)
0925 15:23:35.151897 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.151904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.151933 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:23:35.152019 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.152027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.152116 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.152219 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:35.152833 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.153888 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.155743 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.156886 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.157099 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.157183 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.157458 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.157580 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:23:35.157881 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.151507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.8.226.67:47965)
0925 15:23:35.151722 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.151743 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:35.151873 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:35.152406 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:35.152414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:35.152436 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:35.152605 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.152613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.154016 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.154021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.155577 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.155589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.157084 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.157091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.157283 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.157306 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:23:35.157417 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:23:35.157567 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:23:35.158598 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.152121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:35083 (local address 127.8.226.66:32959)
0925 15:23:35.152315 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.152320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.152731 (+ 411us) server_negotiation.cc:407] Connection header received
0925 15:23:35.152897 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.152901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.153009 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.153082 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:35.153652 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.154583 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.156626 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.157480 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.157665 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.157732 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.158243 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.158365 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:23:35.158632 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.151922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:32959 (local address 127.8.226.67:35083)
0925 15:23:35.152605 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.152625 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:35.152775 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:35.153133 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:35.153138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:35.153163 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:35.153510 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.153521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.154759 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.154762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.156450 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:35.156464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.157669 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.157674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:35.157850 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.157875 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:23:35.158206 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 15:23:35.158332 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":358,"threads_started":1}
I20260925 15:23:35.159040 26953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8e19d3de1f45f6b60754641240035a" candidate_uuid: "dcba398c4fd542aca092e6013a61e253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560e0de098f0400785b82e6f1f15b1eb" is_pre_election: true
I20260925 15:23:35.159575 26953 raft_consensus.cc:2454] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcba398c4fd542aca092e6013a61e253 in term 0.
I20260925 15:23:35.159628 27084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8e19d3de1f45f6b60754641240035a" candidate_uuid: "dcba398c4fd542aca092e6013a61e253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d76a0abfe749fd880b11465d743921" is_pre_election: true
I20260925 15:23:35.160140 27084 raft_consensus.cc:2454] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcba398c4fd542aca092e6013a61e253 in term 0.
I20260925 15:23:35.160254 27148 leader_election.cc:302] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 560e0de098f0400785b82e6f1f15b1eb, dcba398c4fd542aca092e6013a61e253; no voters:
I20260925 15:23:35.160677 27284 raft_consensus.cc:2781] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:35.160830 27284 raft_consensus.cc:482] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:35.160954 27284 raft_consensus.cc:3037] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:35.163282 27284 raft_consensus.cc:504] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.164000 27284 leader_election.cc:288] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [CANDIDATE]: Term 1 election: Requested vote from peers 560e0de098f0400785b82e6f1f15b1eb (127.8.226.65:43757), c2d76a0abfe749fd880b11465d743921 (127.8.226.66:32959)
I20260925 15:23:35.164526 26953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8e19d3de1f45f6b60754641240035a" candidate_uuid: "dcba398c4fd542aca092e6013a61e253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560e0de098f0400785b82e6f1f15b1eb"
I20260925 15:23:35.164733 26953 raft_consensus.cc:3037] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:35.164909 27084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8e19d3de1f45f6b60754641240035a" candidate_uuid: "dcba398c4fd542aca092e6013a61e253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d76a0abfe749fd880b11465d743921"
I20260925 15:23:35.165174 27084 raft_consensus.cc:3037] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:35.167049 26953 raft_consensus.cc:2454] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcba398c4fd542aca092e6013a61e253 in term 1.
I20260925 15:23:35.167505 27148 leader_election.cc:302] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 560e0de098f0400785b82e6f1f15b1eb, dcba398c4fd542aca092e6013a61e253; no voters:
I20260925 15:23:35.167536 27084 raft_consensus.cc:2454] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcba398c4fd542aca092e6013a61e253 in term 1.
I20260925 15:23:35.167877 27284 raft_consensus.cc:2781] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:35.168344 27284 raft_consensus.cc:686] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 1 LEADER]: Becoming Leader. State: Replica: dcba398c4fd542aca092e6013a61e253, State: Running, Role: LEADER
I20260925 15:23:35.168882 27284 consensus_queue.cc:237] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } }
I20260925 15:23:35.173482 26811 catalog_manager.cc:5495] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcba398c4fd542aca092e6013a61e253 (127.8.226.67). New cstate: current_term: 1 leader_uuid: "dcba398c4fd542aca092e6013a61e253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcba398c4fd542aca092e6013a61e253" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 } health_report { overall_health: UNKNOWN } } }
W20260925 15:23:35.208334 27260 tablet.cc:2369] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:35.218211 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.212929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.8.226.67:40729)
0925 15:23:35.213149 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.213154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.213172 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:23:35.213396 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.213400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.213494 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.213628 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:35.213959 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.214712 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.216174 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.216576 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.216680 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.217180 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:35.217692 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:35.217802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.217946 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.218009 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 15:23:35.226233 27084 raft_consensus.cc:1261] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 1 FOLLOWER]: Refusing update from remote peer dcba398c4fd542aca092e6013a61e253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:35.226320 26953 raft_consensus.cc:1261] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 1 FOLLOWER]: Refusing update from remote peer dcba398c4fd542aca092e6013a61e253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:35.227340 27287 consensus_queue.cc:1035] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d76a0abfe749fd880b11465d743921" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 32959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:35.227861 27284 consensus_queue.cc:1035] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560e0de098f0400785b82e6f1f15b1eb" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:35.238234 27290 mvcc.cc:205] Tried to move back new op lower bound from 7333272843157151744 to 7333272842936606720. Current Snapshot: MvccSnapshot[applied={T|T < 7333272843157151744}]
I20260925 15:23:35.239032 27291 mvcc.cc:205] Tried to move back new op lower bound from 7333272843157151744 to 7333272842936606720. Current Snapshot: MvccSnapshot[applied={T|T < 7333272843157151744}]
I20260925 15:23:35.253929 26812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:35.254220 26812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:35.257953 26812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f13d631587545ddb303e9f76ffbc3ae: Sending DeleteTablet for 3 replicas of tablet ea8e19d3de1f45f6b60754641240035a
I20260925 15:23:35.259105 27194 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8e19d3de1f45f6b60754641240035a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:35 UTC) from {username='slave'} at 127.0.0.1:46878
I20260925 15:23:35.259275 26933 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8e19d3de1f45f6b60754641240035a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:35 UTC) from {username='slave'} at 127.0.0.1:51316
I20260925 15:23:35.259418 27064 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8e19d3de1f45f6b60754641240035a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:35 UTC) from {username='slave'} at 127.0.0.1:37942
I20260925 15:23:35.261042 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26871
I20260925 15:23:35.270812 27301 tablet_replica.cc:330] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: stopping tablet replica
I20260925 15:23:35.271384 27301 raft_consensus.cc:2229] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:35.272087 27301 raft_consensus.cc:2258] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:35.274490 27301 ts_tablet_manager.cc:1905] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:35.275197 27300 tablet_replica.cc:330] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: stopping tablet replica
I20260925 15:23:35.275738 27300 raft_consensus.cc:2229] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:35.277419 27300 raft_consensus.cc:2258] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:35.279942 27300 ts_tablet_manager.cc:1905] T ea8e19d3de1f45f6b60754641240035a P 560e0de098f0400785b82e6f1f15b1eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:35.286566 27301 ts_tablet_manager.cc:1918] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:35.286773 27301 log.cc:1196] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/ea8e19d3de1f45f6b60754641240035a
I20260925 15:23:35.287338 27301 ts_tablet_manager.cc:1939] T ea8e19d3de1f45f6b60754641240035a P c2d76a0abfe749fd880b11465d743921: Deleting consensus metadata
I20260925 15:23:35.287457 27299 tablet_replica.cc:330] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: stopping tablet replica
I20260925 15:23:35.287863 27299 raft_consensus.cc:2229] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:35.288434 27299 raft_consensus.cc:2258] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:35.280575 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.262639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.8.226.65:43757)
0925 15:23:35.262862 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.262871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.265538 (+ 2667us) server_negotiation.cc:407] Connection header received
0925 15:23:35.267342 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.267353 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.267492 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.267585 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:35.268470 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.270106 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.275116 (+ 5010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.276154 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.276301 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.278289 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:35.278305 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:35.278329 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:35.279355 (+ 1026us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:35.279435 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:35.279468 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:35.279578 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.279915 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.280089 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:23:35.289402 26797 catalog_manager.cc:4841] TS c2d76a0abfe749fd880b11465d743921 (127.8.226.66:32959): tablet ea8e19d3de1f45f6b60754641240035a (table test-table [id=d1286be6211a4be99ccf3a4a9881ff51]) successfully deleted
I20260925 15:23:35.289583 26993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:35.291608 27299 ts_tablet_manager.cc:1905] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:35.297269 27299 ts_tablet_manager.cc:1918] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:35.297451 27299 log.cc:1196] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/ea8e19d3de1f45f6b60754641240035a
I20260925 15:23:35.297911 27299 ts_tablet_manager.cc:1939] T ea8e19d3de1f45f6b60754641240035a P dcba398c4fd542aca092e6013a61e253: Deleting consensus metadata
I20260925 15:23:35.299515 26799 catalog_manager.cc:4841] TS dcba398c4fd542aca092e6013a61e253 (127.8.226.67:40729): tablet ea8e19d3de1f45f6b60754641240035a (table test-table [id=d1286be6211a4be99ccf3a4a9881ff51]) successfully deleted
I20260925 15:23:35.411042 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26871
W20260925 15:23:35.428252 26797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:43757 (error 108)
I20260925 15:23:35.428511 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27002
I20260925 15:23:35.430667 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.430256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51872)
0925 15:23:35.430421 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.430534 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:23:35.430989 26797 catalog_manager.cc:4584] TS 560e0de098f0400785b82e6f1f15b1eb (127.8.226.65:43757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea8e19d3de1f45f6b60754641240035a: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
I20260925 15:23:35.436100 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.429520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.8.226.66:32959)
0925 15:23:35.429725 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.429730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.429751 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:23:35.429818 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.429822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.429911 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.429995 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:35.430534 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.431697 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.433170 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.433588 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.433742 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.434807 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:35.434816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:35.434829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:35.435396 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:35.435441 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:35.435460 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:35.435578 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.435704 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.435810 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:23:35.436873 27124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:35.484717 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.484136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51882)
0925 15:23:35.484384 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.484494 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 15:23:35.485935 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.485515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51886)
0925 15:23:35.485679 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.485786 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:23:35.528393 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27002
I20260925 15:23:35.546038 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27132
I20260925 15:23:35.546672 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.546169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51892)
0925 15:23:35.546405 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.546510 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:23:35.547837 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.547479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51900)
0925 15:23:35.547622 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.547699 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:23:35.554224 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.546772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.8.226.67:40729)
0925 15:23:35.546943 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:35.546950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:35.547180 (+ 230us) server_negotiation.cc:407] Connection header received
0925 15:23:35.547314 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:35.547322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:35.547446 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:35.547559 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:35.548134 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.549383 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.550662 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:35.551367 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:35.551508 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:35.552512 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:35.552523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:35.552540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:35.553427 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:35.553493 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:35.553521 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:35.553648 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:35.553758 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:23:35.553854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:23:35.554988 27254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:35.628682 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.628123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51908)
0925 15:23:35.628357 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.628472 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:23:35.629819 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.629401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51912)
0925 15:23:35.629605 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.629691 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 15:23:35.676242 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27132
I20260925 15:23:35.695713 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 26781
I20260925 15:23:35.696862 26842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:35.805778 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:35.805171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:43757 (local address 127.0.0.1:51914)
0925 15:23:35.805491 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:35.805603 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 15:23:35.805794 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 26781
2026-09-25T15:23:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2774 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:35 dist-test-slave-8vw7 krb5kdc[27310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:35 dist-test-slave-8vw7 krb5kdc[27310](info): set up 2 sockets
Sep 25 15:23:35 dist-test-slave-8vw7 krb5kdc[27310](info): commencing operation
krb5kdc: starting...
W20260925 15:23:36.917284 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:37 dist-test-slave-8vw7 krb5kdc[27310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:23:37.166487 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:37597
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:37597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:37.493132 27323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:37.493423 27323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:37.493602 27323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:37.500825 27323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:37.500943 27323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:37.501050 27323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:37.501145 27323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:37.501226 27323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:37.511025 27323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:37597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:37.512323 27323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:37.513690 27323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:37.525652 27331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.525718 27328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.525736 27329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:37.526549 27323 server_base.cc:1034] running on GCE node
I20260925 15:23:37.527354 27323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:37.528730 27323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:37.529939 27323 hybrid_clock.cc:648] HybridClock initialized: now 1790349817529888 us; error 39 us; skew 500 ppm
I20260925 15:23:37.530267 27323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:37 dist-test-slave-8vw7 krb5kdc[27310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:37.533360 27323 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:23:37.534639 27323 webserver.cc:460] Webserver started at http://127.8.226.126:40919/ using document root <none> and password file <none>
I20260925 15:23:37.535226 27323 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:37.535354 27323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:37.535595 27323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:37.538506 27323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "75ff17156eb94afe87e4c32d9fd4cf7c"
format_stamp: "Formatted at 2026-09-25 15:23:37 on dist-test-slave-8vw7"
server_key: "de0b55871a9a29aa9061b0aa575f44c9"
server_key_iv: "ef49032e576bf7102abd6bd0d101ed08"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.539247 27323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "75ff17156eb94afe87e4c32d9fd4cf7c"
format_stamp: "Formatted at 2026-09-25 15:23:37 on dist-test-slave-8vw7"
server_key: "de0b55871a9a29aa9061b0aa575f44c9"
server_key_iv: "ef49032e576bf7102abd6bd0d101ed08"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.544071 27323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:23:37.547473 27338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.548738 27323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:37.548866 27323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "75ff17156eb94afe87e4c32d9fd4cf7c"
format_stamp: "Formatted at 2026-09-25 15:23:37 on dist-test-slave-8vw7"
server_key: "de0b55871a9a29aa9061b0aa575f44c9"
server_key_iv: "ef49032e576bf7102abd6bd0d101ed08"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.549094 27323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:37.566741 27323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:37.571081 27323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:37.571312 27323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:37.589490 27323 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:37597
I20260925 15:23:37.589512 27389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:37597 every 8 connection(s)
I20260925 15:23:37.590909 27323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:37.594530 27390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:37.600081 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27323
I20260925 15:23:37.600273 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:37.600711 9097 external_mini_cluster.cc:1428] Setting key f4217fad30b00380ba4b9a807d756ee3
I20260925 15:23:37.605187 27390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c: Bootstrap starting.
I20260925 15:23:37.608677 27390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:37.609916 27390 log.cc:824] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:37.612574 27390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c: No bootstrap required, opened a new log
Sep 25 15:23:37 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:37.621687 27390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37597 } }
I20260925 15:23:37.622152 27390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:37.622260 27390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ff17156eb94afe87e4c32d9fd4cf7c, State: Initialized, Role: FOLLOWER
I20260925 15:23:37.622845 27390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37597 } }
I20260925 15:23:37.623088 27390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:37.623221 27390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:37.623394 27390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:37.623581 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.602532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.8.226.126:37597)
0925 15:23:37.603406 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.603417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.603454 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:23:37.605176 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.605198 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.605599 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.605912 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:37.607714 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.609137 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.610922 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.611561 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.611765 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.616182 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:37.616189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:37.616202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:37.616246 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:37.619326 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.620162 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.620168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.620173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.620336 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.620742 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.620744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.620746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.621271 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:37.621487 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.621951 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.622402 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":440,"threads_started":1}
I20260925 15:23:37.626300 27390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37597 } }
I20260925 15:23:37.626828 27390 leader_election.cc:302] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ff17156eb94afe87e4c32d9fd4cf7c; no voters:
I20260925 15:23:37.627354 27390 leader_election.cc:288] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:37.627485 27395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:37.628171 27395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [term 1 LEADER]: Becoming Leader. State: Replica: 75ff17156eb94afe87e4c32d9fd4cf7c, State: Running, Role: LEADER
I20260925 15:23:37.628649 27395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37597 } }
I20260925 15:23:37.629443 27390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:37.631352 27396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37597 } } }
I20260925 15:23:37.631649 27396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:37.632287 27397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ff17156eb94afe87e4c32d9fd4cf7c. Latest consensus state: current_term: 1 leader_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ff17156eb94afe87e4c32d9fd4cf7c" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 37597 } } }
I20260925 15:23:37.632484 27401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:37.632570 27397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:37.640240 27401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:37.648612 27401 catalog_manager.cc:1347] Generated new cluster ID: debfb7a6c5ef4c31b011b35fe580994c
I20260925 15:23:37.648743 27401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:37.663785 27401 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:37.665073 27401 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:37.674553 27401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c: Generated new TSK 0
I20260925 15:23:37.675390 27401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:23:37.731637 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37597
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:38.064752 27416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:38.065104 27416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:38.065250 27416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:38.072453 27416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:38.072570 27416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:38.072733 27416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:38.082578 27416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:38.083913 27416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:38.085274 27416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:38.098677 27421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:38.098785 27424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:38.098781 27422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:38.099083 27416 server_base.cc:1034] running on GCE node
I20260925 15:23:38.099879 27416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:38.100808 27416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:38.102029 27416 hybrid_clock.cc:648] HybridClock initialized: now 1790349818101979 us; error 61 us; skew 500 ppm
I20260925 15:23:38.102342 27416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:38 dist-test-slave-8vw7 krb5kdc[27310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:38.105398 27416 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:23:38.106834 27416 webserver.cc:460] Webserver started at http://127.8.226.65:41413/ using document root <none> and password file <none>
I20260925 15:23:38.107517 27416 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:38.107653 27416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:38.107962 27416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:38.110926 27416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "4d2308e878d54ff6a1803bdbe1fa909e"
format_stamp: "Formatted at 2026-09-25 15:23:38 on dist-test-slave-8vw7"
server_key: "a649bef509ac1ef42b15e9dfd0906011"
server_key_iv: "e74d2c2ecd3f7a5f8c0f758fe546daff"
server_key_version: "encryptionkey@0"
I20260925 15:23:38.111698 27416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "4d2308e878d54ff6a1803bdbe1fa909e"
format_stamp: "Formatted at 2026-09-25 15:23:38 on dist-test-slave-8vw7"
server_key: "a649bef509ac1ef42b15e9dfd0906011"
server_key_iv: "e74d2c2ecd3f7a5f8c0f758fe546daff"
server_key_version: "encryptionkey@0"
I20260925 15:23:38.116791 27416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 15:23:38.120334 27431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:38.121816 27416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:38.121985 27416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "4d2308e878d54ff6a1803bdbe1fa909e"
format_stamp: "Formatted at 2026-09-25 15:23:38 on dist-test-slave-8vw7"
server_key: "a649bef509ac1ef42b15e9dfd0906011"
server_key_iv: "e74d2c2ecd3f7a5f8c0f758fe546daff"
server_key_version: "encryptionkey@0"
I20260925 15:23:38.122176 27416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:38.141458 27416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:38.145480 27416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:38.145762 27416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:38.146741 27416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:38.148077 27416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:38.148173 27416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:38.148322 27416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:38.148391 27416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:38.180814 27416 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:39049
I20260925 15:23:38.180863 27543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:39049 every 8 connection(s)
I20260925 15:23:38.182119 27416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:38.188777 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27416
I20260925 15:23:38.188998 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:38.189592 9097 external_mini_cluster.cc:1428] Setting key 8c6394df238634de013fc3f5faba4a3b
Sep 25 15:23:38 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:38.205032 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:38.184645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:47395 (local address 127.8.226.126:37597)
0925 15:23:38.185180 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:38.185185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:38.185978 (+ 793us) server_negotiation.cc:407] Connection header received
0925 15:23:38.187808 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:38.187813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:38.187903 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:38.188017 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:38.190190 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.191201 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.194122 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.194539 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.194657 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:38.199077 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:38.199083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:38.199086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:38.199121 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:38.201832 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:38.202630 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:38.202635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:38.202637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:38.202759 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:38.203197 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:38.203203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:38.203214 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:38.203484 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:38.203640 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:38.204407 (+ 767us) server_negotiation.cc:299] Negotiation successful
0925 15:23:38.204640 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":255,"threads_started":1}
I20260925 15:23:38.206024 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:38.185147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37597 (local address 127.8.226.65:47395)
0925 15:23:38.185816 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:38.185853 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:38.187539 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:38.188268 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:38.188276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:38.189092 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:38.189945 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:38.189972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.191399 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.191407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:38.193934 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:38.193955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.194804 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.194812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:38.195155 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:38.195951 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:38.195992 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:38.198803 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:38.201995 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:38.202006 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:38.202025 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:38.202487 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:38.202869 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:38.202874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:38.202878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:38.203078 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:38.203611 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:38.203619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:38.204033 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 15:23:38.204304 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 15:23:38.204879 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":259,"threads_started":1}
I20260925 15:23:38.207581 27544 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37597
I20260925 15:23:38.207973 27544 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:38.208874 27544 heartbeater.cc:511] Master 127.8.226.126:37597 requested a full tablet report, sending...
I20260925 15:23:38.211443 27355 ts_manager.cc:194] Registered new tserver with Master: 4d2308e878d54ff6a1803bdbe1fa909e (127.8.226.65:39049)
I20260925 15:23:38.213008 27355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:47395
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:23:38.251394 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37597
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:38.580403 27550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:38.580689 27550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:38.580785 27550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:38.587987 27550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:38.588097 27550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:38.588215 27550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:38.597855 27550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:38.599086 27550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:38.600363 27550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:38.612872 27558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:38.612913 27555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:38.612993 27556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:38.613802 27550 server_base.cc:1034] running on GCE node
I20260925 15:23:38.614389 27550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:38.615257 27550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:38.616469 27550 hybrid_clock.cc:648] HybridClock initialized: now 1790349818616423 us; error 53 us; skew 500 ppm
I20260925 15:23:38.616770 27550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:38 dist-test-slave-8vw7 krb5kdc[27310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:38.624195 27550 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:23:38.625722 27550 webserver.cc:460] Webserver started at http://127.8.226.66:35403/ using document root <none> and password file <none>
I20260925 15:23:38.626423 27550 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:38.626559 27550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:38.626852 27550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:38.629812 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "c7c0ff0f380a47d9aea305740bc6d461"
format_stamp: "Formatted at 2026-09-25 15:23:38 on dist-test-slave-8vw7"
server_key: "9b0ef89fa8a175e91af232076f69d83e"
server_key_iv: "73ae02ddf53feeaee8213c593e997857"
server_key_version: "encryptionkey@0"
I20260925 15:23:38.630582 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "c7c0ff0f380a47d9aea305740bc6d461"
format_stamp: "Formatted at 2026-09-25 15:23:38 on dist-test-slave-8vw7"
server_key: "9b0ef89fa8a175e91af232076f69d83e"
server_key_iv: "73ae02ddf53feeaee8213c593e997857"
server_key_version: "encryptionkey@0"
I20260925 15:23:38.635676 27550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:23:38.639056 27565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:38.640309 27550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:38.640473 27550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "c7c0ff0f380a47d9aea305740bc6d461"
format_stamp: "Formatted at 2026-09-25 15:23:38 on dist-test-slave-8vw7"
server_key: "9b0ef89fa8a175e91af232076f69d83e"
server_key_iv: "73ae02ddf53feeaee8213c593e997857"
server_key_version: "encryptionkey@0"
I20260925 15:23:38.640682 27550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:38.657262 27550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:38.661121 27550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:38.661405 27550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:38.662340 27550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:38.663637 27550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:38.663743 27550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:38.663885 27550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:38.663954 27550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:38.696620 27550 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:45241
I20260925 15:23:38.696667 27677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:45241 every 8 connection(s)
I20260925 15:23:38.697945 27550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:38.705788 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27550
I20260925 15:23:38.705991 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:38.706418 9097 external_mini_cluster.cc:1428] Setting key b124d2b5828b5fc330d8182d4543f214
Sep 25 15:23:38 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:38.717756 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:38.700399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:38469 (local address 127.8.226.126:37597)
0925 15:23:38.700569 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:38.700573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:38.701735 (+ 1162us) server_negotiation.cc:407] Connection header received
0925 15:23:38.703539 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:38.703544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:38.703632 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:38.703736 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:38.705230 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.706004 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.707713 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.708193 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.708324 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:38.711665 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:38.711676 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:38.711679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:38.711710 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:38.714360 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:38.715106 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:38.715111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:38.715115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:38.715248 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:38.715654 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:38.715659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:38.715662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:38.716004 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:38.716149 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:38.717066 (+ 917us) server_negotiation.cc:299] Negotiation successful
0925 15:23:38.717346 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:23:38.718897 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:38.700885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37597 (local address 127.8.226.66:38469)
0925 15:23:38.701579 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:38.701614 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:38.703284 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:38.703965 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:38.703973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:38.704460 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:38.705070 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:38.705088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.706159 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.706163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:38.707577 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:38.707589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:38.708413 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:38.708420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:38.708632 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:38.709323 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:38.709344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:38.711448 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:38.714517 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:38.714526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:38.714544 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:38.714983 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:38.715352 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:38.715358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:38.715361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:38.715543 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:38.716121 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:38.716129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:38.716559 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 15:23:38.716879 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 15:23:38.717471 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":253,"threads_started":1}
I20260925 15:23:38.720498 27678 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37597
I20260925 15:23:38.720818 27678 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:38.721669 27678 heartbeater.cc:511] Master 127.8.226.126:37597 requested a full tablet report, sending...
I20260925 15:23:38.723248 27355 ts_manager.cc:194] Registered new tserver with Master: c7c0ff0f380a47d9aea305740bc6d461 (127.8.226.66:45241)
I20260925 15:23:38.724063 27355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:38469
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:23:38.762044 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37597
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:39.082582 27683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:39.082885 27683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:39.083050 27683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:39.089948 27683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:39.090078 27683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:39.090253 27683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:39.099606 27683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:37597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:39.100802 27683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:39.102084 27683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:39.114857 27689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:39.114935 27688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:39.114953 27691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:39.115664 27683 server_base.cc:1034] running on GCE node
I20260925 15:23:39.116211 27683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:39.117116 27683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:39.118306 27683 hybrid_clock.cc:648] HybridClock initialized: now 1790349819118261 us; error 38 us; skew 500 ppm
I20260925 15:23:39.118618 27683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:39 dist-test-slave-8vw7 krb5kdc[27310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349819, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:39.121954 27683 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:23:39.123591 27683 webserver.cc:460] Webserver started at http://127.8.226.67:38341/ using document root <none> and password file <none>
I20260925 15:23:39.124290 27683 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:39.124428 27683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:39.124745 27683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:39.127923 27683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "59a84c4647df42728ebc71c2b5eb953b"
format_stamp: "Formatted at 2026-09-25 15:23:39 on dist-test-slave-8vw7"
server_key: "563bf5306f09cc3fd63ee158a18b5929"
server_key_iv: "095da74e8fbfe2e4aa433a37c7dd2efb"
server_key_version: "encryptionkey@0"
I20260925 15:23:39.128674 27683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "59a84c4647df42728ebc71c2b5eb953b"
format_stamp: "Formatted at 2026-09-25 15:23:39 on dist-test-slave-8vw7"
server_key: "563bf5306f09cc3fd63ee158a18b5929"
server_key_iv: "095da74e8fbfe2e4aa433a37c7dd2efb"
server_key_version: "encryptionkey@0"
I20260925 15:23:39.133750 27683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:23:39.137099 27698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:39.138340 27683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:23:39.138499 27683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "59a84c4647df42728ebc71c2b5eb953b"
format_stamp: "Formatted at 2026-09-25 15:23:39 on dist-test-slave-8vw7"
server_key: "563bf5306f09cc3fd63ee158a18b5929"
server_key_iv: "095da74e8fbfe2e4aa433a37c7dd2efb"
server_key_version: "encryptionkey@0"
I20260925 15:23:39.138666 27683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:39.161972 27683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:39.165984 27683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:39.166204 27683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:39.166997 27683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:39.168130 27683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:39.168197 27683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:39.168275 27683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:39.168318 27683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:39.200809 27683 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:40823
I20260925 15:23:39.200863 27810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:40823 every 8 connection(s)
I20260925 15:23:39.202144 27683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:39.204646 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27683
I20260925 15:23:39.204829 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:39.205349 9097 external_mini_cluster.cc:1428] Setting key 7c11df1a4523e615fc14cb728ba17303
Sep 25 15:23:39 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349819, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:39.216806 27544 heartbeater.cc:503] Master 127.8.226.126:37597 was elected leader, sending a full tablet report...
I20260925 15:23:39.222911 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.204778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:57343 (local address 127.8.226.126:37597)
0925 15:23:39.204997 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.205003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.206285 (+ 1282us) server_negotiation.cc:407] Connection header received
0925 15:23:39.208126 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.208131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.208217 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.208327 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:39.209830 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.210568 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.212001 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.212399 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.212487 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.215765 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:39.215770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:39.215773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:39.215805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:39.219405 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.220513 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.220517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.220519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.220647 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.221168 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.221172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.221174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.221445 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:39.221593 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.222334 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.222563 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:23:39.223878 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.205287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:37597 (local address 127.8.226.67:57343)
0925 15:23:39.206125 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:39.206181 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:39.207891 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:39.208518 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:39.208526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:39.209093 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:39.209682 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.209700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.210687 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.210691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.211888 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.211898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.212573 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.212578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.212777 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.213450 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:39.213470 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:39.215574 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:39.219577 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:39.219587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:39.219606 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:39.220245 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:39.220764 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:39.220769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:39.220773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:39.221039 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:39.221587 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:39.221595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:39.221961 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 15:23:39.222226 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 15:23:39.222786 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260925 15:23:39.225317 27811 heartbeater.cc:347] Connected to a master server at 127.8.226.126:37597
I20260925 15:23:39.225610 27811 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:39.226428 27811 heartbeater.cc:511] Master 127.8.226.126:37597 requested a full tablet report, sending...
I20260925 15:23:39.227824 27355 ts_manager.cc:194] Registered new tserver with Master: 59a84c4647df42728ebc71c2b5eb953b (127.8.226.67:40823)
I20260925 15:23:39.228619 27355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:57343
I20260925 15:23:39.232456 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:39.249527 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.236520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.8.226.126:37597)
0925 15:23:39.236715 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.236723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.237185 (+ 462us) server_negotiation.cc:407] Connection header received
0925 15:23:39.237347 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.237352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.237413 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.237489 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:39.238800 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.239517 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.240868 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.241399 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.241492 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.244186 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:39.244191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:39.244194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:39.244225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:39.246899 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.247633 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.247636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.247639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.247754 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.248138 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.248141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.248143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.248431 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:39.248634 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.249026 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.249194 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:23:39.253305 27355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:39.260030 27355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:39.291239 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.271038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39049 (local address 127.0.0.1:51344)
0925 15:23:39.278034 (+ 6996us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:39.278072 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:39.278267 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:39.282645 (+ 4378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:39.282652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:39.282681 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:39.283146 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.283164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.285674 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.285682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.287894 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.287906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.290193 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.290200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.290512 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.290571 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:23:39.290690 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:23:39.290899 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6769,"thread_start_us":6633,"threads_started":1}
I20260925 15:23:39.291411 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.271295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.8.226.65:39049)
0925 15:23:39.281680 (+ 10385us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.281693 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.281744 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:23:39.281881 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.281892 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.282212 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.282421 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:39.283399 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.285476 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.288091 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.289702 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.289846 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.290067 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.290886 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.291064 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10151,"thread_start_us":9954,"threads_started":1}
I20260925 15:23:39.292730 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.271284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:40823 (local address 127.0.0.1:58650)
0925 15:23:39.281973 (+ 10689us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:39.281998 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:39.282109 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:39.283777 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:39.283785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:39.283820 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:39.284371 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.284386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.286575 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.286580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.288992 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.289007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.290770 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.290776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.292133 (+ 1357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.292161 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:23:39.292252 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:23:39.292376 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10542,"mutex_wait_us":1195,"thread_start_us":10415,"threads_started":1}
I20260925 15:23:39.293045 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.271903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.8.226.67:40823)
0925 15:23:39.282720 (+ 10817us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.282732 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.282771 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:23:39.282885 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.282895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.283196 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.283407 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:39.284624 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.286435 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.289147 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.290559 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.290696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.290910 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.292309 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.292479 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10608,"thread_start_us":10436,"threads_started":1}
I20260925 15:23:39.296006 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.282782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:45241 (local address 127.0.0.1:39484)
0925 15:23:39.286332 (+ 3550us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:39.286357 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:39.286477 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:39.287188 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:39.287193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:39.287225 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:39.287724 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.287738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.289754 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.289758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.291853 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.291867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.295236 (+ 3369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.295242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.295404 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.295435 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:23:39.295533 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:23:39.295652 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3412,"mutex_wait_us":1,"thread_start_us":3281,"threads_started":1}
I20260925 15:23:39.296041 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.282357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.8.226.66:45241)
0925 15:23:39.286014 (+ 3657us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.286022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.286595 (+ 573us) server_negotiation.cc:407] Connection header received
0925 15:23:39.286731 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.286739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.286987 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.287930 (+ 943us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:39.288140 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.289595 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.292001 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.295028 (+ 3027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.295173 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.295343 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.295592 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.295761 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":3405,"threads_started":1}
I20260925 15:23:39.295558 27479 tablet_service.cc:1467] Processing CreateTablet for tablet ac610f89ff8f419b8d3764eefafc8537 (DEFAULT_TABLE table=test-table [id=1b8e50d5812c4d1a958d8140bd83fa78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:39.297065 27479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac610f89ff8f419b8d3764eefafc8537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:39.296599 27746 tablet_service.cc:1467] Processing CreateTablet for tablet ac610f89ff8f419b8d3764eefafc8537 (DEFAULT_TABLE table=test-table [id=1b8e50d5812c4d1a958d8140bd83fa78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:39.298060 27746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac610f89ff8f419b8d3764eefafc8537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:39.299606 27613 tablet_service.cc:1467] Processing CreateTablet for tablet ac610f89ff8f419b8d3764eefafc8537 (DEFAULT_TABLE table=test-table [id=1b8e50d5812c4d1a958d8140bd83fa78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:39.301251 27613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac610f89ff8f419b8d3764eefafc8537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:39.308015 27829 tablet_bootstrap.cc:492] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Bootstrap starting.
I20260925 15:23:39.310935 27829 tablet_bootstrap.cc:654] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:39.312285 27830 tablet_bootstrap.cc:492] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Bootstrap starting.
I20260925 15:23:39.312400 27829 log.cc:824] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:39.315461 27829 tablet_bootstrap.cc:492] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: No bootstrap required, opened a new log
I20260925 15:23:39.315762 27829 ts_tablet_manager.cc:1397] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260925 15:23:39.315771 27830 tablet_bootstrap.cc:654] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:39.316084 27832 tablet_bootstrap.cc:492] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Bootstrap starting.
I20260925 15:23:39.316998 27830 log.cc:824] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:39.319463 27832 tablet_bootstrap.cc:654] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:39.319880 27830 tablet_bootstrap.cc:492] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: No bootstrap required, opened a new log
I20260925 15:23:39.320127 27830 ts_tablet_manager.cc:1397] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:23:39.320675 27832 log.cc:824] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:39.323042 27832 tablet_bootstrap.cc:492] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: No bootstrap required, opened a new log
I20260925 15:23:39.323304 27832 ts_tablet_manager.cc:1397] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:23:39.326118 27829 raft_consensus.cc:348] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.326539 27829 raft_consensus.cc:374] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:39.326633 27829 raft_consensus.cc:729] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2308e878d54ff6a1803bdbe1fa909e, State: Initialized, Role: FOLLOWER
I20260925 15:23:39.327239 27829 consensus_queue.cc:260] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.330355 27829 ts_tablet_manager.cc:1428] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260925 15:23:39.331036 27830 raft_consensus.cc:348] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.331466 27830 raft_consensus.cc:374] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:39.331561 27830 raft_consensus.cc:729] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a84c4647df42728ebc71c2b5eb953b, State: Initialized, Role: FOLLOWER
I20260925 15:23:39.332140 27830 consensus_queue.cc:260] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.333434 27811 heartbeater.cc:503] Master 127.8.226.126:37597 was elected leader, sending a full tablet report...
I20260925 15:23:39.333452 27832 raft_consensus.cc:348] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.333880 27832 raft_consensus.cc:374] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:39.333972 27832 raft_consensus.cc:729] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c0ff0f380a47d9aea305740bc6d461, State: Initialized, Role: FOLLOWER
I20260925 15:23:39.334008 27830 ts_tablet_manager.cc:1428] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260925 15:23:39.334648 27832 consensus_queue.cc:260] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.336395 27678 heartbeater.cc:503] Master 127.8.226.126:37597 was elected leader, sending a full tablet report...
I20260925 15:23:39.337041 27832 ts_tablet_manager.cc:1428] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260925 15:23:39.434453 27545 tablet.cc:2369] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:23:39.449783 27679 tablet.cc:2369] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:23:39.453679 27812 tablet.cc:2369] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:39.541106 27836 raft_consensus.cc:482] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:39.541499 27836 raft_consensus.cc:504] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.543346 27836 leader_election.cc:288] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d2308e878d54ff6a1803bdbe1fa909e (127.8.226.65:39049), c7c0ff0f380a47d9aea305740bc6d461 (127.8.226.66:45241)
I20260925 15:23:39.550906 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.544369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:47449 (local address 127.8.226.66:45241)
0925 15:23:39.544581 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.544586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.544607 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:23:39.544680 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.544689 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.544789 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.544879 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:39.545383 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.546391 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.548724 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.550001 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.550164 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.550242 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.550520 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.550649 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:23:39.550940 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.543793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:45241 (local address 127.8.226.67:47449)
0925 15:23:39.543984 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:39.544005 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:39.544139 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:39.544943 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:39.544974 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:39.544997 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:39.545225 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.545240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.546517 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.546520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.548569 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.548583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.550152 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.550156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.550356 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.550396 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:23:39.550493 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:23:39.550630 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260925 15:23:39.551182 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.543793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:39049 (local address 127.8.226.67:50459)
0925 15:23:39.544656 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:39.544672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:39.544787 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:39.545259 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:39.545263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:39.545292 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:39.545568 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.545575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.546795 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.546800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.548792 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:39.548801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.550380 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.550385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:39.550551 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.550579 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:23:39.550652 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:23:39.550767 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":42,"thread_start_us":451,"threads_started":1}
I20260925 15:23:39.551223 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.543873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:50459 (local address 127.8.226.65:39049)
0925 15:23:39.544088 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.544096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.544757 (+ 661us) server_negotiation.cc:407] Connection header received
0925 15:23:39.545047 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.545050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.545111 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.545243 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:39.545714 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.546677 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.548936 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.550191 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.550328 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.550376 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.550689 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.550822 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:23:39.552016 27633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac610f89ff8f419b8d3764eefafc8537" candidate_uuid: "59a84c4647df42728ebc71c2b5eb953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c0ff0f380a47d9aea305740bc6d461" is_pre_election: true
I20260925 15:23:39.552409 27499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac610f89ff8f419b8d3764eefafc8537" candidate_uuid: "59a84c4647df42728ebc71c2b5eb953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" is_pre_election: true
I20260925 15:23:39.552624 27633 raft_consensus.cc:2454] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59a84c4647df42728ebc71c2b5eb953b in term 0.
I20260925 15:23:39.552985 27499 raft_consensus.cc:2454] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59a84c4647df42728ebc71c2b5eb953b in term 0.
I20260925 15:23:39.553378 27700 leader_election.cc:302] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a84c4647df42728ebc71c2b5eb953b, c7c0ff0f380a47d9aea305740bc6d461; no voters:
I20260925 15:23:39.553805 27836 raft_consensus.cc:2781] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:39.553946 27836 raft_consensus.cc:482] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:39.554056 27836 raft_consensus.cc:3037] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:39.556180 27836 raft_consensus.cc:504] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.556731 27836 leader_election.cc:288] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [CANDIDATE]: Term 1 election: Requested vote from peers 4d2308e878d54ff6a1803bdbe1fa909e (127.8.226.65:39049), c7c0ff0f380a47d9aea305740bc6d461 (127.8.226.66:45241)
I20260925 15:23:39.557161 27633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac610f89ff8f419b8d3764eefafc8537" candidate_uuid: "59a84c4647df42728ebc71c2b5eb953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c0ff0f380a47d9aea305740bc6d461"
I20260925 15:23:39.557210 27499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac610f89ff8f419b8d3764eefafc8537" candidate_uuid: "59a84c4647df42728ebc71c2b5eb953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2308e878d54ff6a1803bdbe1fa909e"
I20260925 15:23:39.557358 27633 raft_consensus.cc:3037] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:39.557421 27499 raft_consensus.cc:3037] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:39.559446 27633 raft_consensus.cc:2454] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59a84c4647df42728ebc71c2b5eb953b in term 1.
I20260925 15:23:39.559587 27499 raft_consensus.cc:2454] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59a84c4647df42728ebc71c2b5eb953b in term 1.
I20260925 15:23:39.559842 27700 leader_election.cc:302] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a84c4647df42728ebc71c2b5eb953b, c7c0ff0f380a47d9aea305740bc6d461; no voters:
I20260925 15:23:39.560094 27836 raft_consensus.cc:2781] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:39.560536 27836 raft_consensus.cc:686] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 1 LEADER]: Becoming Leader. State: Replica: 59a84c4647df42728ebc71c2b5eb953b, State: Running, Role: LEADER
I20260925 15:23:39.561008 27836 consensus_queue.cc:237] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } }
I20260925 15:23:39.565269 27355 catalog_manager.cc:5495] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b reported cstate change: term changed from 0 to 1, leader changed from <none> to 59a84c4647df42728ebc71c2b5eb953b (127.8.226.67). New cstate: current_term: 1 leader_uuid: "59a84c4647df42728ebc71c2b5eb953b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59a84c4647df42728ebc71c2b5eb953b" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 40823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 } health_report { overall_health: UNKNOWN } } }
I20260925 15:23:39.594525 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.588122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.8.226.67:40823)
0925 15:23:39.588301 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.588306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.588353 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:23:39.588539 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.588543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.588649 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.588779 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:39.589271 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.590030 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.592011 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.592410 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.592506 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.593242 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:39.593819 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:39.593970 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.594143 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.594212 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:23:39.603632 27633 raft_consensus.cc:1261] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 1 FOLLOWER]: Refusing update from remote peer 59a84c4647df42728ebc71c2b5eb953b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:39.603785 27499 raft_consensus.cc:1261] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 1 FOLLOWER]: Refusing update from remote peer 59a84c4647df42728ebc71c2b5eb953b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:39.604733 27836 consensus_queue.cc:1035] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d2308e878d54ff6a1803bdbe1fa909e" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:39.605096 27839 consensus_queue.cc:1035] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c0ff0f380a47d9aea305740bc6d461" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:39.615456 27842 mvcc.cc:205] Tried to move back new op lower bound from 7333272861086449664 to 7333272860925362176. Current Snapshot: MvccSnapshot[applied={T|T < 7333272861086449664}]
I20260925 15:23:39.621915 27845 mvcc.cc:205] Tried to move back new op lower bound from 7333272861086449664 to 7333272860925362176. Current Snapshot: MvccSnapshot[applied={T|T < 7333272861086449664}]
I20260925 15:23:39.632385 27354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:39.632656 27354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:39.636759 27354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ff17156eb94afe87e4c32d9fd4cf7c: Sending DeleteTablet for 3 replicas of tablet ac610f89ff8f419b8d3764eefafc8537
I20260925 15:23:39.638631 27479 tablet_service.cc:1514] Processing DeleteTablet for tablet ac610f89ff8f419b8d3764eefafc8537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:39 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:51344
I20260925 15:23:39.638631 27746 tablet_service.cc:1514] Processing DeleteTablet for tablet ac610f89ff8f419b8d3764eefafc8537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:39 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:58650
I20260925 15:23:39.639593 27613 tablet_service.cc:1514] Processing DeleteTablet for tablet ac610f89ff8f419b8d3764eefafc8537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:39 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:39484
I20260925 15:23:39.640057 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27416
I20260925 15:23:39.651873 27855 tablet_replica.cc:330] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: stopping tablet replica
I20260925 15:23:39.652406 27855 raft_consensus.cc:2229] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:39.652946 27855 raft_consensus.cc:2258] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:39.654672 27854 tablet_replica.cc:330] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: stopping tablet replica
I20260925 15:23:39.655067 27854 raft_consensus.cc:2229] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:39.655534 27854 raft_consensus.cc:2258] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:39.659044 27855 ts_tablet_manager.cc:1905] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:39.659389 27854 ts_tablet_manager.cc:1905] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:39.666535 27855 ts_tablet_manager.cc:1918] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:39.666838 27855 log.cc:1196] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/ac610f89ff8f419b8d3764eefafc8537
I20260925 15:23:39.667470 27855 ts_tablet_manager.cc:1939] T ac610f89ff8f419b8d3764eefafc8537 P c7c0ff0f380a47d9aea305740bc6d461: Deleting consensus metadata
Sep 25 15:23:39 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:23:39.669771 27340 catalog_manager.cc:4841] TS c7c0ff0f380a47d9aea305740bc6d461 (127.8.226.66:45241): tablet ac610f89ff8f419b8d3764eefafc8537 (table test-table [id=1b8e50d5812c4d1a958d8140bd83fa78]) successfully deleted
I20260925 15:23:39.670902 27853 tablet_replica.cc:330] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: stopping tablet replica
I20260925 15:23:39.671418 27853 raft_consensus.cc:2229] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:39.672103 27853 raft_consensus.cc:2258] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:39.675130 27853 ts_tablet_manager.cc:1905] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:39.677461 27854 ts_tablet_manager.cc:1918] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:39.677657 27854 log.cc:1196] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/ac610f89ff8f419b8d3764eefafc8537
I20260925 15:23:39.678246 27854 ts_tablet_manager.cc:1939] T ac610f89ff8f419b8d3764eefafc8537 P 4d2308e878d54ff6a1803bdbe1fa909e: Deleting consensus metadata
I20260925 15:23:39.679376 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.641377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.8.226.65:39049)
0925 15:23:39.641551 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.641558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.641634 (+ 76us) server_negotiation.cc:407] Connection header received
0925 15:23:39.641953 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.641959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.642043 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.642129 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:39.644464 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.646209 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.648985 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.650072 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.650213 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.670448 (+ 20235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:39.670469 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:39.670490 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:39.670552 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:39.674750 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.675807 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.675818 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.675822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.676004 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.676492 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.676498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.676502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.677035 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:39.677337 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.678688 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.678961 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:23:39.680291 27539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:39.680317 27341 catalog_manager.cc:4841] TS 4d2308e878d54ff6a1803bdbe1fa909e (127.8.226.65:39049): tablet ac610f89ff8f419b8d3764eefafc8537 (table test-table [id=1b8e50d5812c4d1a958d8140bd83fa78]) successfully deleted
I20260925 15:23:39.690327 27853 ts_tablet_manager.cc:1918] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:39.690557 27853 log.cc:1196] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/ac610f89ff8f419b8d3764eefafc8537
I20260925 15:23:39.691231 27853 ts_tablet_manager.cc:1939] T ac610f89ff8f419b8d3764eefafc8537 P 59a84c4647df42728ebc71c2b5eb953b: Deleting consensus metadata
I20260925 15:23:39.693579 27340 catalog_manager.cc:4841] TS 59a84c4647df42728ebc71c2b5eb953b (127.8.226.67:40823): tablet ac610f89ff8f419b8d3764eefafc8537 (table test-table [id=1b8e50d5812c4d1a958d8140bd83fa78]) successfully deleted
I20260925 15:23:39.814075 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27416
I20260925 15:23:39.831243 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27550
Sep 25 15:23:39 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:23:39.848382 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.832021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.8.226.66:45241)
0925 15:23:39.832224 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.832230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.832336 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:23:39.832507 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.832515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.832601 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.832716 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:39.834018 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.835242 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.836681 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.837334 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.837461 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.841830 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:39.841849 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:39.841871 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:39.841915 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:39.845654 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.846332 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.846342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.846347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.846500 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.846858 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.846863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.846867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.847309 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:39.847582 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.847825 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.848033 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:23:39.849056 27673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:39.937011 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27550
I20260925 15:23:39.953883 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27683
Sep 25 15:23:39 dist-test-slave-8vw7 krb5kdc[27310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:23:39.969837 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:39.954630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.8.226.67:40823)
0925 15:23:39.954854 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:39.954861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:39.954945 (+ 84us) server_negotiation.cc:407] Connection header received
0925 15:23:39.955154 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:39.955162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:39.955234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:39.955343 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:39.956618 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.957554 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.958794 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:39.959274 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:39.959405 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:39.964021 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:39.964032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:39.964049 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:39.964085 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:39.967002 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.967677 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.967689 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.967692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.967830 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:39.968196 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:39.968206 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:39.968208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:39.968559 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:39.968873 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:39.969240 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:23:39.969455 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 15:23:39.970566 27806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:40.093921 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27683
I20260925 15:23:40.109274 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27323
I20260925 15:23:40.110400 27385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:40.219223 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27323
2026-09-25T15:23:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4413 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:23:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:40Z Disabled control of system clock
I20260925 15:23:40.293571 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39677
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:39677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:23:40.619549 27864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:40.619828 27864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:40.619907 27864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:40.627096 27864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:40.627195 27864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:40.627239 27864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:40.627295 27864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:40.627338 27864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:40.636227 27864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:39677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:40.637392 27864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:40.639081 27864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:40.650318 27869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:40.650398 27872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:40.650368 27870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:40.651310 27864 server_base.cc:1034] running on GCE node
I20260925 15:23:40.652040 27864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:40.653489 27864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:40.654698 27864 hybrid_clock.cc:648] HybridClock initialized: now 1790349820654651 us; error 33 us; skew 500 ppm
I20260925 15:23:40.655041 27864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:40.657239 27864 webserver.cc:460] Webserver started at http://127.8.226.126:40619/ using document root <none> and password file <none>
I20260925 15:23:40.657902 27864 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:40.657987 27864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:40.658313 27864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:40.661157 27864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "2958ff170cde45879844d3dd7e0d4707"
format_stamp: "Formatted at 2026-09-25 15:23:40 on dist-test-slave-8vw7"
server_key: "5b896b04b42db30c141a4d73ded54daa"
server_key_iv: "3e24c759e0df6976e99c8ec3c2e4851c"
server_key_version: "encryptionkey@0"
I20260925 15:23:40.661952 27864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "2958ff170cde45879844d3dd7e0d4707"
format_stamp: "Formatted at 2026-09-25 15:23:40 on dist-test-slave-8vw7"
server_key: "5b896b04b42db30c141a4d73ded54daa"
server_key_iv: "3e24c759e0df6976e99c8ec3c2e4851c"
server_key_version: "encryptionkey@0"
I20260925 15:23:40.666879 27864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:23:40.670235 27878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:40.671671 27864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:23:40.671864 27864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "2958ff170cde45879844d3dd7e0d4707"
format_stamp: "Formatted at 2026-09-25 15:23:40 on dist-test-slave-8vw7"
server_key: "5b896b04b42db30c141a4d73ded54daa"
server_key_iv: "3e24c759e0df6976e99c8ec3c2e4851c"
server_key_version: "encryptionkey@0"
I20260925 15:23:40.672052 27864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:40.683969 27864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:40.684677 27864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:40.684988 27864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:40.702502 27864 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:39677
I20260925 15:23:40.702507 27929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:39677 every 8 connection(s)
I20260925 15:23:40.703872 27864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:40.706686 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27864
I20260925 15:23:40.706876 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:40.707317 9097 external_mini_cluster.cc:1428] Setting key 71a3412e9e0799263e306759f4ff6780
I20260925 15:23:40.707562 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:40.722388 27933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:40.709191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.8.226.126:39677)
0925 15:23:40.710232 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:40.710249 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:40.710302 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:23:40.712364 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:40.712390 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:40.713009 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:40.713455 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:40.713995 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:40.716024 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:40.717440 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:40.718458 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:40.718635 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:40.719689 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:40.719701 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:40.719722 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:40.720449 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:40.720529 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:40.720553 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:40.720660 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:40.720880 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:23:40.721192 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":338,"threads_started":1}
I20260925 15:23:40.722687 27930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707: Bootstrap starting.
I20260925 15:23:40.725800 27930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:40.726722 27930 log.cc:824] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:40.729184 27930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707: No bootstrap required, opened a new log
I20260925 15:23:40.736130 27930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958ff170cde45879844d3dd7e0d4707" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39677 } }
I20260925 15:23:40.736461 27930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:40.736567 27930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2958ff170cde45879844d3dd7e0d4707, State: Initialized, Role: FOLLOWER
I20260925 15:23:40.737174 27930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958ff170cde45879844d3dd7e0d4707" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39677 } }
I20260925 15:23:40.737421 27930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:40.737543 27930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:40.737712 27930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:40.739795 27930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958ff170cde45879844d3dd7e0d4707" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39677 } }
I20260925 15:23:40.740257 27930 leader_election.cc:302] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2958ff170cde45879844d3dd7e0d4707; no voters:
I20260925 15:23:40.740765 27930 leader_election.cc:288] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:40.740882 27935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:40.741617 27935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [term 1 LEADER]: Becoming Leader. State: Replica: 2958ff170cde45879844d3dd7e0d4707, State: Running, Role: LEADER
I20260925 15:23:40.742082 27935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958ff170cde45879844d3dd7e0d4707" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39677 } }
I20260925 15:23:40.742714 27930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:40.744686 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2958ff170cde45879844d3dd7e0d4707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958ff170cde45879844d3dd7e0d4707" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39677 } } }
I20260925 15:23:40.745254 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:40.744843 27937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2958ff170cde45879844d3dd7e0d4707. Latest consensus state: current_term: 1 leader_uuid: "2958ff170cde45879844d3dd7e0d4707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2958ff170cde45879844d3dd7e0d4707" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39677 } } }
I20260925 15:23:40.745486 27937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:40.747052 27941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:40.752170 27941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:40.760579 27941 catalog_manager.cc:1347] Generated new cluster ID: 477be59cf79c41e6b28e09162e3ae3f6
I20260925 15:23:40.760679 27941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:40.795612 27941 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:40.796875 27941 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:40.802636 27941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707: Generated new TSK 0
I20260925 15:23:40.803623 27941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:40.821421 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39677
--builtin_ntp_servers=127.8.226.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:23:41.150362 27954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:41.150622 27954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:41.150760 27954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:41.157763 27954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:41.157850 27954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:41.158043 27954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:41.166911 27954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:41.168133 27954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:41.170011 27954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:41.182627 27959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:41.182715 27960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:41.182693 27962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:41.183559 27954 server_base.cc:1034] running on GCE node
I20260925 15:23:41.184094 27954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:41.185025 27954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:41.186241 27954 hybrid_clock.cc:648] HybridClock initialized: now 1790349821186189 us; error 43 us; skew 500 ppm
I20260925 15:23:41.186528 27954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:41.188750 27954 webserver.cc:460] Webserver started at http://127.8.226.65:34079/ using document root <none> and password file <none>
I20260925 15:23:41.189445 27954 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:41.189576 27954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:41.189886 27954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:41.192557 27954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "7f4761e8e6664c50b696b87e6954373b"
format_stamp: "Formatted at 2026-09-25 15:23:41 on dist-test-slave-8vw7"
server_key: "5b6e85946356e72f0d714c239f4b60c4"
server_key_iv: "7196636e2390d0895011265abc6324e8"
server_key_version: "encryptionkey@0"
I20260925 15:23:41.193357 27954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "7f4761e8e6664c50b696b87e6954373b"
format_stamp: "Formatted at 2026-09-25 15:23:41 on dist-test-slave-8vw7"
server_key: "5b6e85946356e72f0d714c239f4b60c4"
server_key_iv: "7196636e2390d0895011265abc6324e8"
server_key_version: "encryptionkey@0"
I20260925 15:23:41.197989 27954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:23:41.201309 27968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:41.202522 27954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:23:41.202685 27954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "7f4761e8e6664c50b696b87e6954373b"
format_stamp: "Formatted at 2026-09-25 15:23:41 on dist-test-slave-8vw7"
server_key: "5b6e85946356e72f0d714c239f4b60c4"
server_key_iv: "7196636e2390d0895011265abc6324e8"
server_key_version: "encryptionkey@0"
I20260925 15:23:41.202890 27954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:41.216054 27954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:41.216817 27954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:41.217163 27954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:41.218068 27954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:41.219446 27954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:41.219563 27954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:41.219693 27954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:41.219779 27954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:41.251616 27954 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:46271
I20260925 15:23:41.251662 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:46271 every 8 connection(s)
I20260925 15:23:41.252868 27954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:41.257239 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 27954
I20260925 15:23:41.257393 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:41.257809 9097 external_mini_cluster.cc:1428] Setting key 7144afbe497ccd05275b6609b5614aee
I20260925 15:23:41.260612 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39677
--builtin_ntp_servers=127.8.226.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:23:41.266701 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:41.255362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:41917 (local address 127.8.226.126:39677)
0925 15:23:41.255786 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:41.255791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:41.256970 (+ 1179us) server_negotiation.cc:407] Connection header received
0925 15:23:41.258986 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:41.258992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:41.259078 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:41.259191 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:41.260482 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.261393 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.263951 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.264327 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.264414 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:41.266004 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:41.266009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:41.266012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:41.266029 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:41.266065 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:41.266076 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:41.266143 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:41.266391 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 15:23:41.266464 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":234,"threads_started":1}
I20260925 15:23:41.268092 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:41.255964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39677 (local address 127.8.226.65:41917)
0925 15:23:41.256802 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:41.256841 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:41.258729 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:41.259412 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:41.259419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:41.259529 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:41.260306 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:41.260334 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.261532 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.261540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:41.263812 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:41.263831 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.264539 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.264548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:41.264873 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:41.265665 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:41.265671 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:41.265725 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:41.265822 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:41.266170 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:41.266184 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:41.266195 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:23:41.266372 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 15:23:41.266769 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":259,"threads_started":1}
I20260925 15:23:41.269614 28081 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39677
I20260925 15:23:41.269930 28081 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:41.270790 28081 heartbeater.cc:511] Master 127.8.226.126:39677 requested a full tablet report, sending...
I20260925 15:23:41.272708 27895 ts_manager.cc:194] Registered new tserver with Master: 7f4761e8e6664c50b696b87e6954373b (127.8.226.65:46271)
I20260925 15:23:41.274228 27895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.65:41917
W20260925 15:23:41.588899 28085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:41.589233 28085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:41.589373 28085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:41.596581 28085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:41.596701 28085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:41.596880 28085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:41.605693 28085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:41.606869 28085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:41.608583 28085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:41.621065 28091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:41.621197 28090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:41.621192 28093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:41.621986 28085 server_base.cc:1034] running on GCE node
I20260925 15:23:41.622484 28085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:41.623314 28085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:41.624514 28085 hybrid_clock.cc:648] HybridClock initialized: now 1790349821624448 us; error 64 us; skew 500 ppm
I20260925 15:23:41.624763 28085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:41.626981 28085 webserver.cc:460] Webserver started at http://127.8.226.66:35041/ using document root <none> and password file <none>
I20260925 15:23:41.627555 28085 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:41.627662 28085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:41.627910 28085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:41.630560 28085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "dcc78192c5354fc6a14d33b239ecee89"
format_stamp: "Formatted at 2026-09-25 15:23:41 on dist-test-slave-8vw7"
server_key: "52b90e64aadff16d8a0a1811be29fca1"
server_key_iv: "e752459586bb972bdd1dda4e0e159bc0"
server_key_version: "encryptionkey@0"
I20260925 15:23:41.631253 28085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "dcc78192c5354fc6a14d33b239ecee89"
format_stamp: "Formatted at 2026-09-25 15:23:41 on dist-test-slave-8vw7"
server_key: "52b90e64aadff16d8a0a1811be29fca1"
server_key_iv: "e752459586bb972bdd1dda4e0e159bc0"
server_key_version: "encryptionkey@0"
I20260925 15:23:41.635826 28085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:23:41.638890 28099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:41.640062 28085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 15:23:41.640215 28085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "dcc78192c5354fc6a14d33b239ecee89"
format_stamp: "Formatted at 2026-09-25 15:23:41 on dist-test-slave-8vw7"
server_key: "52b90e64aadff16d8a0a1811be29fca1"
server_key_iv: "e752459586bb972bdd1dda4e0e159bc0"
server_key_version: "encryptionkey@0"
I20260925 15:23:41.640414 28085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:41.670068 28085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:41.670964 28085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:41.671283 28085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:41.672287 28085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:41.673866 28085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:41.673982 28085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:41.674110 28085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:41.674196 28085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:41.707085 28085 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:44451
I20260925 15:23:41.707295 28211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:44451 every 8 connection(s)
I20260925 15:23:41.708420 28085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:41.717423 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28085
I20260925 15:23:41.717602 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:41.718147 9097 external_mini_cluster.cc:1428] Setting key 7893244e80f5db47a020323b9403d68b
I20260925 15:23:41.721017 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39677
--builtin_ntp_servers=127.8.226.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:23:41.723513 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:41.711088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:60343 (local address 127.8.226.126:39677)
0925 15:23:41.711308 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:41.711315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:41.712821 (+ 1506us) server_negotiation.cc:407] Connection header received
0925 15:23:41.715401 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:41.715406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:41.715481 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:41.715581 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:41.716897 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.718083 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.720328 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.721008 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.721145 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:41.722571 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:41.722580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:41.722587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:41.722612 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:41.722656 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:41.722673 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:41.722780 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:41.723025 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:23:41.723138 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:23:41.724450 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:41.711585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39677 (local address 127.8.226.66:60343)
0925 15:23:41.712625 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:41.712679 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:41.715051 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:41.715822 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:41.715834 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:41.715948 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:41.716727 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:41.716751 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.718245 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.718251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:41.720173 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:41.720188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:41.721275 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:41.721282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:41.721509 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:41.722258 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:41.722261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:41.722301 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:41.722377 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:41.722806 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:41.722815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:41.722822 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:23:41.722983 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 15:23:41.723305 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":277,"threads_started":1}
I20260925 15:23:41.726034 28212 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39677
I20260925 15:23:41.726369 28212 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:41.727245 28212 heartbeater.cc:511] Master 127.8.226.126:39677 requested a full tablet report, sending...
I20260925 15:23:41.728818 27895 ts_manager.cc:194] Registered new tserver with Master: dcc78192c5354fc6a14d33b239ecee89 (127.8.226.66:44451)
I20260925 15:23:41.730007 27895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.66:60343
W20260925 15:23:42.046573 28215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:42.046870 28215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:42.047026 28215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:42.054267 28215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:42.054380 28215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:42.054545 28215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:42.063238 28215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:42.064399 28215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:42.066201 28215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:42.078799 28220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:42.078840 28223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:42.078861 28221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:42.079531 28215 server_base.cc:1034] running on GCE node
I20260925 15:23:42.080076 28215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:42.080920 28215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:42.082155 28215 hybrid_clock.cc:648] HybridClock initialized: now 1790349822082117 us; error 31 us; skew 500 ppm
I20260925 15:23:42.082440 28215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:42.084462 28215 webserver.cc:460] Webserver started at http://127.8.226.67:43927/ using document root <none> and password file <none>
I20260925 15:23:42.085157 28215 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:42.085269 28215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:42.085570 28215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:42.088157 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "d377a2b17f8543ec8d625b0e0f9dfa27"
format_stamp: "Formatted at 2026-09-25 15:23:42 on dist-test-slave-8vw7"
server_key: "c7a2cca586005b9f01751e9d01a8a476"
server_key_iv: "7fd4fc79cacbde387148233e9e6c54f1"
server_key_version: "encryptionkey@0"
I20260925 15:23:42.088891 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "d377a2b17f8543ec8d625b0e0f9dfa27"
format_stamp: "Formatted at 2026-09-25 15:23:42 on dist-test-slave-8vw7"
server_key: "c7a2cca586005b9f01751e9d01a8a476"
server_key_iv: "7fd4fc79cacbde387148233e9e6c54f1"
server_key_version: "encryptionkey@0"
I20260925 15:23:42.093530 28215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:23:42.096726 28229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:42.098044 28215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:23:42.098202 28215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "d377a2b17f8543ec8d625b0e0f9dfa27"
format_stamp: "Formatted at 2026-09-25 15:23:42 on dist-test-slave-8vw7"
server_key: "c7a2cca586005b9f01751e9d01a8a476"
server_key_iv: "7fd4fc79cacbde387148233e9e6c54f1"
server_key_version: "encryptionkey@0"
I20260925 15:23:42.098399 28215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:42.110266 28215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:42.110976 28215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:42.111248 28215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:42.112102 28215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:42.113416 28215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:42.113521 28215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:42.113647 28215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:42.113718 28215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:42.145337 28215 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:41261
I20260925 15:23:42.145401 28341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:41261 every 8 connection(s)
I20260925 15:23:42.146618 28215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:42.147169 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28215
I20260925 15:23:42.147351 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:42.148011 9097 external_mini_cluster.cc:1428] Setting key ed88e68fac2a71b52b5f34b72b828e5c
I20260925 15:23:42.158891 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.149234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:57019 (local address 127.8.226.126:39677)
0925 15:23:42.149421 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.149426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.151073 (+ 1647us) server_negotiation.cc:407] Connection header received
0925 15:23:42.152764 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.152769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.152855 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.152952 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:42.153974 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.154732 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.156236 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.156651 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.156769 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.158110 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:42.158116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:42.158121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:42.158144 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:42.158179 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:42.158191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:42.158279 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.158491 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.158581 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:23:42.159776 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.149845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:39677 (local address 127.8.226.67:57019)
0925 15:23:42.150929 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.150974 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.152519 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.153189 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.153197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.153278 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:42.153803 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.153818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.154870 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.154875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.156100 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.156111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.156861 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.156868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.157151 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.157801 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:42.157803 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:42.157838 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:42.157917 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:42.158278 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:42.158289 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:42.158298 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.158465 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.158821 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":302,"threads_started":1}
I20260925 15:23:42.161103 28342 heartbeater.cc:347] Connected to a master server at 127.8.226.126:39677
I20260925 15:23:42.161365 28342 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:42.162248 28342 heartbeater.cc:511] Master 127.8.226.126:39677 requested a full tablet report, sending...
I20260925 15:23:42.163573 27895 ts_manager.cc:194] Registered new tserver with Master: d377a2b17f8543ec8d625b0e0f9dfa27 (127.8.226.67:41261)
I20260925 15:23:42.164330 27895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.67:57019
I20260925 15:23:42.174925 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:42.184880 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.178898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.8.226.126:39677)
0925 15:23:42.179084 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.179093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.179447 (+ 354us) server_negotiation.cc:407] Connection header received
0925 15:23:42.179669 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.179675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.179762 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.179856 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:42.180322 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.181096 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.182525 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.182935 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.183069 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.184153 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:42.184165 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:42.184170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:42.184187 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:42.184218 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:42.184230 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:42.184381 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.184560 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.184621 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:23:42.188300 27895 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:42.194880 27895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:42.224838 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.206158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.8.226.65:46271)
0925 15:23:42.216648 (+ 10490us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.216667 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.216691 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:23:42.216868 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.216876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.217148 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.217356 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.217978 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.219518 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.222405 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.223734 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.223879 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.224118 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.224329 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.224483 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10354,"thread_start_us":10266,"threads_started":1}
I20260925 15:23:42.224819 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.205922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43258)
0925 15:23:42.216414 (+ 10492us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.216466 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.216625 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.217354 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.217361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.217397 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.217804 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.217818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.219690 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.219695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.222207 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.222223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.223904 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.223910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.224086 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.224132 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.224281 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.224450 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9690,"mutex_wait_us":22,"thread_start_us":10335,"threads_started":1}
I20260925 15:23:42.226627 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.206490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.8.226.66:44451)
0925 15:23:42.215431 (+ 8941us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.215441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.217244 (+ 1803us) server_negotiation.cc:407] Connection header received
0925 15:23:42.217998 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.218008 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.218249 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.218424 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.219248 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.220751 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.223460 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.225423 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.225548 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.225780 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.226275 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.226390 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8729,"thread_start_us":8556,"threads_started":1}
I20260925 15:23:42.226747 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.206147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:44451 (local address 127.0.0.1:53982)
0925 15:23:42.217024 (+ 10877us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.217040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.217782 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.218671 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.218676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.218700 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.219058 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.219070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.220914 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.220920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.223243 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.223257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.225911 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.225918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.226104 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.226136 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.226225 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.226356 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10712,"mutex_wait_us":28,"thread_start_us":9276,"threads_started":1}
I20260925 15:23:42.229773 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.217112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:41261 (local address 127.0.0.1:43690)
0925 15:23:42.221572 (+ 4460us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.221592 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.221719 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.222383 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.222389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.222418 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.222792 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.222801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.224801 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.224810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.227170 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.227181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.229024 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.229031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.229211 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.229243 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.229336 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.229451 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4343,"mutex_wait_us":20,"thread_start_us":4378,"threads_started":1}
I20260925 15:23:42.229902 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.217465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.8.226.67:41261)
0925 15:23:42.221471 (+ 4006us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.221478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.221845 (+ 367us) server_negotiation.cc:407] Connection header received
0925 15:23:42.221956 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.221964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.222183 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.222940 (+ 757us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.223150 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.224635 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.227585 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.228809 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.228970 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.229243 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.229389 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.229544 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3839,"thread_start_us":3418,"threads_started":1}
I20260925 15:23:42.230497 28147 tablet_service.cc:1467] Processing CreateTablet for tablet bb86348cf8df4392bc4c6093fe4b909e (DEFAULT_TABLE table=test-table [id=2f10a9f6a65447db8ee77bfedcb63e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:42.231098 28016 tablet_service.cc:1467] Processing CreateTablet for tablet bb86348cf8df4392bc4c6093fe4b909e (DEFAULT_TABLE table=test-table [id=2f10a9f6a65447db8ee77bfedcb63e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:42.231895 28147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb86348cf8df4392bc4c6093fe4b909e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:42.232234 28016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb86348cf8df4392bc4c6093fe4b909e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:42.233446 28277 tablet_service.cc:1467] Processing CreateTablet for tablet bb86348cf8df4392bc4c6093fe4b909e (DEFAULT_TABLE table=test-table [id=2f10a9f6a65447db8ee77bfedcb63e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:42.234740 28277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb86348cf8df4392bc4c6093fe4b909e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:42.243160 28360 tablet_bootstrap.cc:492] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: Bootstrap starting.
I20260925 15:23:42.246872 28360 tablet_bootstrap.cc:654] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:42.247982 28360 log.cc:824] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:42.249166 28362 tablet_bootstrap.cc:492] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Bootstrap starting.
I20260925 15:23:42.252571 28361 tablet_bootstrap.cc:492] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Bootstrap starting.
I20260925 15:23:42.254311 28362 tablet_bootstrap.cc:654] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:42.255676 28362 log.cc:824] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:42.256985 28361 tablet_bootstrap.cc:654] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:42.257632 28360 tablet_bootstrap.cc:492] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: No bootstrap required, opened a new log
I20260925 15:23:42.258040 28360 ts_tablet_manager.cc:1397] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260925 15:23:42.258131 28361 log.cc:824] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:42.267052 28361 tablet_bootstrap.cc:492] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: No bootstrap required, opened a new log
I20260925 15:23:42.267343 28361 ts_tablet_manager.cc:1397] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260925 15:23:42.267365 28362 tablet_bootstrap.cc:492] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: No bootstrap required, opened a new log
I20260925 15:23:42.267596 28362 ts_tablet_manager.cc:1397] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20260925 15:23:42.271569 28360 raft_consensus.cc:348] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.271991 28360 raft_consensus.cc:374] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:42.272085 28360 raft_consensus.cc:729] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4761e8e6664c50b696b87e6954373b, State: Initialized, Role: FOLLOWER
I20260925 15:23:42.272694 28360 consensus_queue.cc:260] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.274060 28081 heartbeater.cc:503] Master 127.8.226.126:39677 was elected leader, sending a full tablet report...
I20260925 15:23:42.274718 28360 ts_tablet_manager.cc:1428] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260925 15:23:42.276479 28361 raft_consensus.cc:348] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.276885 28361 raft_consensus.cc:374] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:42.276990 28361 raft_consensus.cc:729] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc78192c5354fc6a14d33b239ecee89, State: Initialized, Role: FOLLOWER
I20260925 15:23:42.277531 28361 consensus_queue.cc:260] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.278004 28362 raft_consensus.cc:348] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.278402 28362 raft_consensus.cc:374] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:42.278517 28362 raft_consensus.cc:729] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d377a2b17f8543ec8d625b0e0f9dfa27, State: Initialized, Role: FOLLOWER
I20260925 15:23:42.279063 28362 consensus_queue.cc:260] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.280197 28342 heartbeater.cc:503] Master 127.8.226.126:39677 was elected leader, sending a full tablet report...
I20260925 15:23:42.280794 28362 ts_tablet_manager.cc:1428] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260925 15:23:42.285633 28212 heartbeater.cc:503] Master 127.8.226.126:39677 was elected leader, sending a full tablet report...
I20260925 15:23:42.286063 28361 ts_tablet_manager.cc:1428] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20260925 15:23:42.298316 28368 raft_consensus.cc:482] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:42.298553 28368 raft_consensus.cc:504] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.300314 28368 leader_election.cc:288] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f4761e8e6664c50b696b87e6954373b (127.8.226.65:46271), dcc78192c5354fc6a14d33b239ecee89 (127.8.226.66:44451)
I20260925 15:23:42.300876 28367 raft_consensus.cc:482] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:42.301203 28367 raft_consensus.cc:504] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.303164 28367 leader_election.cc:288] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f4761e8e6664c50b696b87e6954373b (127.8.226.65:46271), d377a2b17f8543ec8d625b0e0f9dfa27 (127.8.226.67:41261)
I20260925 15:23:42.309401 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.300464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.8.226.67:39357)
0925 15:23:42.300664 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.300685 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.300801 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.301503 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.301509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.301537 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.301738 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.301765 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.303159 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.303164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.305836 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.305857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.308218 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.308227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.308501 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.308555 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.308662 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.308858 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":2}
I20260925 15:23:42.310092 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.301545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:35089 (local address 127.8.226.66:44451)
0925 15:23:42.301733 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.301738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.301788 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:23:42.301855 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.301863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.301941 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.302032 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.302579 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.303595 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.306946 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.308672 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.308827 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.308929 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.309680 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.309870 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:23:42.310201 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.300707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:39357 (local address 127.8.226.65:46271)
0925 15:23:42.300946 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.300975 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.301001 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:23:42.301092 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.301101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.301200 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.301295 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.301923 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.302966 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.306082 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.307739 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.307942 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.308045 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.309658 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.309842 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:23:42.310654 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.300623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:44451 (local address 127.8.226.67:35089)
0925 15:23:42.301548 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.301568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.301694 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.302046 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.302050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.302071 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.302397 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.302405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.303834 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.303843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.306682 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.306702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.309132 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.309141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.309417 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.309485 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.310237 (+ 752us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.310376 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":535,"threads_started":1}
I20260925 15:23:42.312898 28036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86348cf8df4392bc4c6093fe4b909e" candidate_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4761e8e6664c50b696b87e6954373b" is_pre_election: true
I20260925 15:23:42.313550 28036 raft_consensus.cc:2454] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d377a2b17f8543ec8d625b0e0f9dfa27 in term 0.
I20260925 15:23:42.313410 28167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86348cf8df4392bc4c6093fe4b909e" candidate_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc78192c5354fc6a14d33b239ecee89" is_pre_election: true
I20260925 15:23:42.313901 28167 raft_consensus.cc:2454] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d377a2b17f8543ec8d625b0e0f9dfa27 in term 0.
I20260925 15:23:42.314293 28232 leader_election.cc:302] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f4761e8e6664c50b696b87e6954373b, d377a2b17f8543ec8d625b0e0f9dfa27; no voters:
I20260925 15:23:42.314703 28368 raft_consensus.cc:2781] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:42.314844 28368 raft_consensus.cc:482] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:42.314951 28368 raft_consensus.cc:3037] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:42.317366 28368 raft_consensus.cc:504] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.318070 28368 leader_election.cc:288] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [CANDIDATE]: Term 1 election: Requested vote from peers 7f4761e8e6664c50b696b87e6954373b (127.8.226.65:46271), dcc78192c5354fc6a14d33b239ecee89 (127.8.226.66:44451)
I20260925 15:23:42.318678 28167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86348cf8df4392bc4c6093fe4b909e" candidate_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc78192c5354fc6a14d33b239ecee89"
I20260925 15:23:42.318701 28036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86348cf8df4392bc4c6093fe4b909e" candidate_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4761e8e6664c50b696b87e6954373b"
I20260925 15:23:42.318876 28167 raft_consensus.cc:3037] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:42.318917 28036 raft_consensus.cc:3037] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:42.320199 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.311516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:57463 (local address 127.8.226.67:41261)
0925 15:23:42.311903 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.311912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.312887 (+ 975us) server_negotiation.cc:407] Connection header received
0925 15:23:42.313151 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.313157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.313242 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.313320 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.314076 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.315135 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.317779 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.318975 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.319133 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.319215 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.319828 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.319958 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260925 15:23:42.320384 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.303641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:38595 (local address 127.8.226.65:46271)
0925 15:23:42.310435 (+ 6794us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.310443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.311100 (+ 657us) server_negotiation.cc:407] Connection header received
0925 15:23:42.311436 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.311445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.311540 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.311614 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:42.312720 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.314858 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.317818 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.319028 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.319195 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.319273 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.319933 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.320075 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6665,"thread_start_us":8274,"threads_started":1}
I20260925 15:23:42.320942 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.311398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:41261 (local address 127.8.226.66:57463)
0925 15:23:42.312793 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.312814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.313005 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.313446 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.313449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.313468 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.313883 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.313895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.315280 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.315283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.317558 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.317572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.319363 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.319372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.319622 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.319670 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.320407 (+ 737us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.320555 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":24,"thread_start_us":1062,"threads_started":1}
I20260925 15:23:42.321471 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.303300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.8.226.66:38595)
0925 15:23:42.310985 (+ 7685us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.311009 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:42.311245 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:42.312021 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:42.312026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:42.312057 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:42.312531 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.312548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.315040 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.315044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.317561 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:42.317572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.319405 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.319412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:42.319740 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.319776 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:23:42.320763 (+ 987us) client_negotiation.cc:240] Negotiation successful
0925 15:23:42.320921 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7509,"mutex_wait_us":248,"thread_start_us":7626,"threads_started":1}
I20260925 15:23:42.322048 28297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86348cf8df4392bc4c6093fe4b909e" candidate_uuid: "dcc78192c5354fc6a14d33b239ecee89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" is_pre_election: true
I20260925 15:23:42.322139 28035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb86348cf8df4392bc4c6093fe4b909e" candidate_uuid: "dcc78192c5354fc6a14d33b239ecee89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4761e8e6664c50b696b87e6954373b" is_pre_election: true
I20260925 15:23:42.322366 28167 raft_consensus.cc:2454] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d377a2b17f8543ec8d625b0e0f9dfa27 in term 1.
I20260925 15:23:42.322393 28036 raft_consensus.cc:2454] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d377a2b17f8543ec8d625b0e0f9dfa27 in term 1.
I20260925 15:23:42.322566 28297 raft_consensus.cc:2379] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dcc78192c5354fc6a14d33b239ecee89 in current term 1: Already voted for candidate d377a2b17f8543ec8d625b0e0f9dfa27 in this term.
I20260925 15:23:42.322747 28231 leader_election.cc:302] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d377a2b17f8543ec8d625b0e0f9dfa27, dcc78192c5354fc6a14d33b239ecee89; no voters:
I20260925 15:23:42.323118 28368 raft_consensus.cc:2781] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:42.323601 28368 raft_consensus.cc:686] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 1 LEADER]: Becoming Leader. State: Replica: d377a2b17f8543ec8d625b0e0f9dfa27, State: Running, Role: LEADER
I20260925 15:23:42.323657 28102 leader_election.cc:302] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcc78192c5354fc6a14d33b239ecee89; no voters: 7f4761e8e6664c50b696b87e6954373b, d377a2b17f8543ec8d625b0e0f9dfa27
I20260925 15:23:42.324167 28367 raft_consensus.cc:2726] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:23:42.324213 28368 consensus_queue.cc:237] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } }
I20260925 15:23:42.329157 27895 catalog_manager.cc:5495] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 reported cstate change: term changed from 0 to 1, leader changed from <none> to d377a2b17f8543ec8d625b0e0f9dfa27 (127.8.226.67). New cstate: current_term: 1 leader_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d377a2b17f8543ec8d625b0e0f9dfa27" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41261 } health_report { overall_health: HEALTHY } } }
I20260925 15:23:42.366618 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.361694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.8.226.67:41261)
0925 15:23:42.361848 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.361852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.361870 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:23:42.362031 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.362035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.362149 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.362272 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:42.362585 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.363343 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.364642 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.365096 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.365195 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.365655 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:42.366142 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:42.366266 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.366370 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.366404 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:23:42.374788 28035 raft_consensus.cc:1261] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 1 FOLLOWER]: Refusing update from remote peer d377a2b17f8543ec8d625b0e0f9dfa27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:42.374842 28167 raft_consensus.cc:1261] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 1 FOLLOWER]: Refusing update from remote peer d377a2b17f8543ec8d625b0e0f9dfa27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:42.375819 28373 consensus_queue.cc:1035] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc78192c5354fc6a14d33b239ecee89" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:42.376225 28368 consensus_queue.cc:1035] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f4761e8e6664c50b696b87e6954373b" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:42.387244 28376 mvcc.cc:205] Tried to move back new op lower bound from 7333272872436707328 to 7333272872244383744. Current Snapshot: MvccSnapshot[applied={T|T < 7333272872436707328}]
I20260925 15:23:42.387941 28377 mvcc.cc:205] Tried to move back new op lower bound from 7333272872436707328 to 7333272872244383744. Current Snapshot: MvccSnapshot[applied={T|T < 7333272872436707328}]
I20260925 15:23:42.401512 27895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:42.401762 27895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:42.405459 27895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2958ff170cde45879844d3dd7e0d4707: Sending DeleteTablet for 3 replicas of tablet bb86348cf8df4392bc4c6093fe4b909e
I20260925 15:23:42.406558 28147 tablet_service.cc:1514] Processing DeleteTablet for tablet bb86348cf8df4392bc4c6093fe4b909e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:42 UTC) from {username='slave'} at 127.0.0.1:53982
I20260925 15:23:42.406566 28277 tablet_service.cc:1514] Processing DeleteTablet for tablet bb86348cf8df4392bc4c6093fe4b909e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:42 UTC) from {username='slave'} at 127.0.0.1:43690
I20260925 15:23:42.406558 28016 tablet_service.cc:1514] Processing DeleteTablet for tablet bb86348cf8df4392bc4c6093fe4b909e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:42 UTC) from {username='slave'} at 127.0.0.1:43258
I20260925 15:23:42.408543 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27954
I20260925 15:23:42.435178 28388 tablet_replica.cc:330] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: stopping tablet replica
I20260925 15:23:42.435810 28388 raft_consensus.cc:2229] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:42.436510 28388 raft_consensus.cc:2258] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:42.439086 28388 ts_tablet_manager.cc:1905] T bb86348cf8df4392bc4c6093fe4b909e P 7f4761e8e6664c50b696b87e6954373b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:42.439738 28387 tablet_replica.cc:330] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: stopping tablet replica
I20260925 15:23:42.443089 28387 raft_consensus.cc:2229] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:42.443742 28387 raft_consensus.cc:2258] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:42.445719 28386 tablet_replica.cc:330] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: stopping tablet replica
I20260925 15:23:42.446209 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.409926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.8.226.65:46271)
0925 15:23:42.410147 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.410157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.433998 (+ 23841us) server_negotiation.cc:407] Connection header received
0925 15:23:42.434201 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.434209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.434356 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.434498 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:42.435336 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.436873 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.440507 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.441454 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.441609 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.443606 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:42.443621 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:42.443644 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:42.444636 (+ 992us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:42.444709 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:42.444740 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:42.444841 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.445739 (+ 898us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.445844 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:23:42.446216 28386 raft_consensus.cc:2229] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:42.447367 28076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:42.447049 28387 ts_tablet_manager.cc:1905] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:42.447873 28386 raft_consensus.cc:2258] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:42.450474 28386 ts_tablet_manager.cc:1905] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:42.456722 28387 ts_tablet_manager.cc:1918] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:42.456928 28387 log.cc:1196] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/bb86348cf8df4392bc4c6093fe4b909e
I20260925 15:23:42.457605 28387 ts_tablet_manager.cc:1939] T bb86348cf8df4392bc4c6093fe4b909e P dcc78192c5354fc6a14d33b239ecee89: Deleting consensus metadata
I20260925 15:23:42.459807 28386 ts_tablet_manager.cc:1918] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:42.459800 27880 catalog_manager.cc:4841] TS dcc78192c5354fc6a14d33b239ecee89 (127.8.226.66:44451): tablet bb86348cf8df4392bc4c6093fe4b909e (table test-table [id=2f10a9f6a65447db8ee77bfedcb63e9a]) successfully deleted
I20260925 15:23:42.460108 28386 log.cc:1196] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/bb86348cf8df4392bc4c6093fe4b909e
I20260925 15:23:42.460855 28386 ts_tablet_manager.cc:1939] T bb86348cf8df4392bc4c6093fe4b909e P d377a2b17f8543ec8d625b0e0f9dfa27: Deleting consensus metadata
I20260925 15:23:42.462577 27881 catalog_manager.cc:4841] TS d377a2b17f8543ec8d625b0e0f9dfa27 (127.8.226.67:41261): tablet bb86348cf8df4392bc4c6093fe4b909e (table test-table [id=2f10a9f6a65447db8ee77bfedcb63e9a]) successfully deleted
I20260925 15:23:42.548695 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27954
W20260925 15:23:42.574040 27881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:46271 (error 108)
I20260925 15:23:42.574141 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28085
I20260925 15:23:42.576099 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.575631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43276)
0925 15:23:42.575804 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.575927 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 15:23:42.576421 27881 catalog_manager.cc:4584] TS 7f4761e8e6664c50b696b87e6954373b (127.8.226.65:46271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb86348cf8df4392bc4c6093fe4b909e: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
I20260925 15:23:42.581383 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.575229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.8.226.66:44451)
0925 15:23:42.575401 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.575407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.575429 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:23:42.575495 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.575501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.575642 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.575768 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:42.576318 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.577230 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.578466 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.578880 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.579006 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.580071 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:42.580079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:42.580093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:42.580629 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:42.580682 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:42.580700 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:42.580825 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.580944 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.581085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:23:42.581995 28207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:42.630064 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.629507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43282)
0925 15:23:42.629749 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.629858 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:23:42.631186 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.630800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43284)
0925 15:23:42.630957 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.631039 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:23:42.671705 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28085
I20260925 15:23:42.688509 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28215
I20260925 15:23:42.691656 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.691256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43300)
0925 15:23:42.691420 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.691503 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:23:42.693792 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.693428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43302)
0925 15:23:42.693571 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.693647 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:23:42.696517 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.689434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.8.226.67:41261)
0925 15:23:42.689652 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:42.689658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:42.689679 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:23:42.689755 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:42.689781 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:42.689883 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:42.689992 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:42.690444 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.691607 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.692902 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:42.693650 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:42.693784 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:42.694879 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:42.694894 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:42.694913 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:42.695759 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:42.695829 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:42.695856 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:42.695972 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:42.696063 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:23:42.696161 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:23:42.697320 28337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:42.774319 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.773842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43306)
0925 15:23:42.774057 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.774153 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:23:42.775441 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:42.775036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46271 (local address 127.0.0.1:43320)
0925 15:23:42.775221 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:42.775292 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 15:23:42.817951 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28215
I20260925 15:23:42.836477 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 27864
I20260925 15:23:42.837646 27925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:42.944074 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 27864
2026-09-25T15:23:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2723 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:43 dist-test-slave-8vw7 krb5kdc[28397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:43 dist-test-slave-8vw7 krb5kdc[28397](info): set up 2 sockets
Sep 25 15:23:43 dist-test-slave-8vw7 krb5kdc[28397](info): commencing operation
krb5kdc: starting...
W20260925 15:23:44.037365 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:44 dist-test-slave-8vw7 krb5kdc[28397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:23:44.283838 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:46683
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:44.606951 28410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:44.607223 28410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:44.607371 28410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:44.614498 28410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:44.614602 28410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:44.614698 28410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:44.614792 28410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:44.614873 28410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:44.624420 28410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:46683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:44.625722 28410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:44.627033 28410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:44.638339 28416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:44.638332 28418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:44.638331 28415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:44.638953 28410 server_base.cc:1034] running on GCE node
I20260925 15:23:44.639698 28410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:44.641010 28410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:44.642212 28410 hybrid_clock.cc:648] HybridClock initialized: now 1790349824642171 us; error 30 us; skew 500 ppm
I20260925 15:23:44.642540 28410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:44 dist-test-slave-8vw7 krb5kdc[28397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:44.651540 28410 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:23:44.653043 28410 webserver.cc:460] Webserver started at http://127.8.226.126:34395/ using document root <none> and password file <none>
I20260925 15:23:44.653708 28410 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:44.653848 28410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:44.654136 28410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:44.657073 28410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "e433eb94fe8d42d4aa44b00d39444503"
format_stamp: "Formatted at 2026-09-25 15:23:44 on dist-test-slave-8vw7"
server_key: "1cca892dc96b0e940c8364fca48d8233"
server_key_iv: "2ceb609a968f7d5bdc55c2be47057cf3"
server_key_version: "encryptionkey@0"
I20260925 15:23:44.657806 28410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "e433eb94fe8d42d4aa44b00d39444503"
format_stamp: "Formatted at 2026-09-25 15:23:44 on dist-test-slave-8vw7"
server_key: "1cca892dc96b0e940c8364fca48d8233"
server_key_iv: "2ceb609a968f7d5bdc55c2be47057cf3"
server_key_version: "encryptionkey@0"
I20260925 15:23:44.662626 28410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:23:44.665911 28425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:44.667160 28410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:23:44.667338 28410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "e433eb94fe8d42d4aa44b00d39444503"
format_stamp: "Formatted at 2026-09-25 15:23:44 on dist-test-slave-8vw7"
server_key: "1cca892dc96b0e940c8364fca48d8233"
server_key_iv: "2ceb609a968f7d5bdc55c2be47057cf3"
server_key_version: "encryptionkey@0"
I20260925 15:23:44.667546 28410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:44.681260 28410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:44.685019 28410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:44.685282 28410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:44.703480 28410 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:46683
I20260925 15:23:44.703500 28476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:46683 every 8 connection(s)
I20260925 15:23:44.704879 28410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:44.706851 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28410
I20260925 15:23:44.707046 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:44.707513 9097 external_mini_cluster.cc:1428] Setting key 36e0a307e34124be26a94ed68ea7a819
I20260925 15:23:44.709477 28477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:23:44 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:44.722579 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503: Bootstrap starting.
I20260925 15:23:44.725796 28477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:44.726914 28477 log.cc:824] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:44.730139 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503: No bootstrap required, opened a new log
I20260925 15:23:44.730587 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:44.709360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.8.226.126:46683)
0925 15:23:44.710243 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:44.710258 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:44.710305 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:23:44.712477 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:44.712504 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:44.713037 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:44.713467 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:44.714793 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:44.716818 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:44.718091 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:44.719108 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:44.719301 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:44.723541 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:44.723548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:44.723561 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:44.723597 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:44.726530 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:44.727318 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:44.727324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:44.727329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:44.727561 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:44.727935 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:44.727944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:44.727947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:44.728542 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:44.728749 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:44.729267 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 15:23:44.729579 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":408,"threads_started":1}
I20260925 15:23:44.737537 28477 raft_consensus.cc:348] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e433eb94fe8d42d4aa44b00d39444503" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46683 } }
I20260925 15:23:44.737877 28477 raft_consensus.cc:374] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:44.737982 28477 raft_consensus.cc:729] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e433eb94fe8d42d4aa44b00d39444503, State: Initialized, Role: FOLLOWER
I20260925 15:23:44.738528 28477 consensus_queue.cc:260] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e433eb94fe8d42d4aa44b00d39444503" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46683 } }
I20260925 15:23:44.738795 28477 raft_consensus.cc:388] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:44.738958 28477 raft_consensus.cc:482] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:44.739156 28477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:44.741360 28477 raft_consensus.cc:504] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e433eb94fe8d42d4aa44b00d39444503" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46683 } }
I20260925 15:23:44.741760 28477 leader_election.cc:302] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e433eb94fe8d42d4aa44b00d39444503; no voters:
I20260925 15:23:44.742277 28477 leader_election.cc:288] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:44.742403 28482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:44.743079 28482 raft_consensus.cc:686] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [term 1 LEADER]: Becoming Leader. State: Replica: e433eb94fe8d42d4aa44b00d39444503, State: Running, Role: LEADER
I20260925 15:23:44.743597 28482 consensus_queue.cc:237] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e433eb94fe8d42d4aa44b00d39444503" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46683 } }
I20260925 15:23:44.744309 28477 sys_catalog.cc:564] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:44.746511 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e433eb94fe8d42d4aa44b00d39444503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e433eb94fe8d42d4aa44b00d39444503" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46683 } } }
I20260925 15:23:44.746954 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:44.746498 28484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e433eb94fe8d42d4aa44b00d39444503. Latest consensus state: current_term: 1 leader_uuid: "e433eb94fe8d42d4aa44b00d39444503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e433eb94fe8d42d4aa44b00d39444503" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46683 } } }
I20260925 15:23:44.747598 28484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:44.753185 28490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:44.759161 28490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:44.768693 28490 catalog_manager.cc:1347] Generated new cluster ID: 74daa8c94f604d37abd4f00e6a49b355
I20260925 15:23:44.769266 28490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:44.783165 28490 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:44.784458 28490 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:44.791338 28490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503: Generated new TSK 0
I20260925 15:23:44.792105 28490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.226.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
Entry for principal kudu/127.8.226.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab.
I20260925 15:23:44.858678 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46683
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:45.179463 28503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:45.179764 28503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:45.179915 28503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:45.186981 28503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:45.187101 28503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:45.187273 28503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:45.196861 28503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.65.keytab
--principal=kudu/127.8.226.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:45.198108 28503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:45.199426 28503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:45.212007 28509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:45.212056 28511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:45.212086 28508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:45.212939 28503 server_base.cc:1034] running on GCE node
I20260925 15:23:45.213541 28503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:45.214401 28503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:45.215618 28503 hybrid_clock.cc:648] HybridClock initialized: now 1790349825215577 us; error 50 us; skew 500 ppm
I20260925 15:23:45.215902 28503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:45 dist-test-slave-8vw7 krb5kdc[28397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:45.218859 28503 init.cc:377] Logged in from keytab as kudu/127.8.226.65@KRBTEST.COM (short username kudu)
I20260925 15:23:45.220206 28503 webserver.cc:460] Webserver started at http://127.8.226.65:38389/ using document root <none> and password file <none>
I20260925 15:23:45.220834 28503 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:45.221028 28503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:45.221323 28503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:45.224031 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "058a0d9ab5604f5d9495a490b2816efd"
format_stamp: "Formatted at 2026-09-25 15:23:45 on dist-test-slave-8vw7"
server_key: "e894f3f3378173f974806e6087e26017"
server_key_iv: "7489150fbe74d78b5087a98eedf3293d"
server_key_version: "encryptionkey@0"
I20260925 15:23:45.224772 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "058a0d9ab5604f5d9495a490b2816efd"
format_stamp: "Formatted at 2026-09-25 15:23:45 on dist-test-slave-8vw7"
server_key: "e894f3f3378173f974806e6087e26017"
server_key_iv: "7489150fbe74d78b5087a98eedf3293d"
server_key_version: "encryptionkey@0"
I20260925 15:23:45.229647 28503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 15:23:45.233042 28518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:45.234308 28503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:23:45.234462 28503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "058a0d9ab5604f5d9495a490b2816efd"
format_stamp: "Formatted at 2026-09-25 15:23:45 on dist-test-slave-8vw7"
server_key: "e894f3f3378173f974806e6087e26017"
server_key_iv: "7489150fbe74d78b5087a98eedf3293d"
server_key_version: "encryptionkey@0"
I20260925 15:23:45.234655 28503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:45.255779 28503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:45.259624 28503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:45.259881 28503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:45.260766 28503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:45.262077 28503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:45.262172 28503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:45.262305 28503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:45.262379 28503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:45.293497 28503 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:44445
I20260925 15:23:45.293507 28630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:44445 every 8 connection(s)
I20260925 15:23:45.294773 28503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:45.304805 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28503
I20260925 15:23:45.305022 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:45.305539 9097 external_mini_cluster.cc:1428] Setting key c2bed9d91dab59d35eaa444aadc84a3d
Sep 25 15:23:45 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.65@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:45.314910 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:45.297168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:55433 (local address 127.8.226.126:46683)
0925 15:23:45.297559 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:45.297563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:45.298558 (+ 995us) server_negotiation.cc:407] Connection header received
0925 15:23:45.300312 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:45.300319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:45.300404 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:45.300512 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:45.301977 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.302612 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.304110 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.304567 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.304713 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:45.308282 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:45.308291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:45.308299 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:45.308345 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:45.311761 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:45.312432 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:45.312440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:45.312446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:45.312581 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:45.313013 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:45.313020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:45.313026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:45.313397 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:45.313560 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:45.314197 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 15:23:45.314438 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":183,"threads_started":1}
I20260925 15:23:45.315435 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:45.297731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:46683 (local address 127.8.226.65:55433)
0925 15:23:45.298406 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:45.298451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:45.300052 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:45.300736 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:45.300744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:45.301298 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:45.301827 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:45.301842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.302752 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.302761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:45.303945 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:45.303968 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.304809 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.304814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:45.305156 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:45.305848 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:45.305868 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:45.308081 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:45.311879 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:45.311903 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:45.311915 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:45.312307 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:45.312681 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:45.312684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:45.312686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:45.312866 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:45.313512 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:45.313518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:45.313828 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 15:23:45.314038 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 15:23:45.314384 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":254,"threads_started":1}
I20260925 15:23:45.316879 28631 heartbeater.cc:347] Connected to a master server at 127.8.226.126:46683
I20260925 15:23:45.317184 28631 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:45.317849 28631 heartbeater.cc:511] Master 127.8.226.126:46683 requested a full tablet report, sending...
I20260925 15:23:45.319736 28442 ts_manager.cc:194] Registered new tserver with Master: 058a0d9ab5604f5d9495a490b2816efd (127.8.226.65:44445)
I20260925 15:23:45.321189 28442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.65@KRBTEST.COM'} at 127.8.226.65:55433
WARNING: no policy specified for kudu/127.8.226.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
Entry for principal kudu/127.8.226.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab.
I20260925 15:23:45.357549 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46683
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:45.675248 28637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:45.675562 28637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:45.675743 28637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:45.682804 28637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:45.682951 28637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:45.683197 28637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:45.693017 28637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.66.keytab
--principal=kudu/127.8.226.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:45.694324 28637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:45.695673 28637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:45.709292 28645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:45.709288 28643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:45.709651 28642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:45.710464 28637 server_base.cc:1034] running on GCE node
I20260925 15:23:45.711071 28637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:45.712015 28637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:45.713239 28637 hybrid_clock.cc:648] HybridClock initialized: now 1790349825713188 us; error 42 us; skew 500 ppm
I20260925 15:23:45.713546 28637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:45 dist-test-slave-8vw7 krb5kdc[28397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:45.716686 28637 init.cc:377] Logged in from keytab as kudu/127.8.226.66@KRBTEST.COM (short username kudu)
I20260925 15:23:45.718137 28637 webserver.cc:460] Webserver started at http://127.8.226.66:41935/ using document root <none> and password file <none>
I20260925 15:23:45.718812 28637 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:45.718936 28637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:45.719247 28637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:45.722086 28637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "b39fa0fcdcee42cdb4477d31322aaedf"
format_stamp: "Formatted at 2026-09-25 15:23:45 on dist-test-slave-8vw7"
server_key: "96807da5f52315ff9a6462ccb6cd6868"
server_key_iv: "91df3554d6b83b9b7bb811d597677c4a"
server_key_version: "encryptionkey@0"
I20260925 15:23:45.722856 28637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "b39fa0fcdcee42cdb4477d31322aaedf"
format_stamp: "Formatted at 2026-09-25 15:23:45 on dist-test-slave-8vw7"
server_key: "96807da5f52315ff9a6462ccb6cd6868"
server_key_iv: "91df3554d6b83b9b7bb811d597677c4a"
server_key_version: "encryptionkey@0"
I20260925 15:23:45.727828 28637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:23:45.731240 28652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:45.732484 28637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:23:45.732654 28637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "b39fa0fcdcee42cdb4477d31322aaedf"
format_stamp: "Formatted at 2026-09-25 15:23:45 on dist-test-slave-8vw7"
server_key: "96807da5f52315ff9a6462ccb6cd6868"
server_key_iv: "91df3554d6b83b9b7bb811d597677c4a"
server_key_version: "encryptionkey@0"
I20260925 15:23:45.732841 28637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:45.745227 28637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:45.748793 28637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:45.749102 28637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:45.750005 28637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:45.751205 28637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:45.751282 28637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:45.751446 28637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:45.751498 28637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:45.783510 28637 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:41549
I20260925 15:23:45.783613 28764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:41549 every 8 connection(s)
I20260925 15:23:45.784792 28637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:45.787815 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28637
I20260925 15:23:45.788028 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:45.788683 9097 external_mini_cluster.cc:1428] Setting key bcaa578fdf093fd5b04e48e69ce74242
Sep 25 15:23:45 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.66@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:45.806885 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:45.787233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:50257 (local address 127.8.226.126:46683)
0925 15:23:45.787451 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:45.787457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:45.788729 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 15:23:45.790919 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:45.790923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:45.790985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:45.791066 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:45.793224 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.793934 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.796319 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.796754 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.796849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:45.801264 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:45.801269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:45.801272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:45.801305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:45.803849 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:45.804566 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:45.804576 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:45.804578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:45.804706 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:45.805222 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:45.805233 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:45.805236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:45.805489 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:45.805643 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:45.806300 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 15:23:45.806525 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:23:45.808038 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:45.787714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:46683 (local address 127.8.226.66:50257)
0925 15:23:45.788544 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:45.788597 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:45.790624 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:45.791352 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:45.791366 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:45.792152 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:45.793034 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:45.793062 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.794088 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.794095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:45.796159 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:45.796185 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:45.797005 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:45.797015 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:45.797355 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:45.798089 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:45.798122 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:45.801042 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:45.803994 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:45.804003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:45.804019 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:45.804438 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:45.804830 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:45.804836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:45.804840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:45.805098 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:45.805614 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:45.805621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:45.805957 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 15:23:45.806207 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 15:23:45.806722 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":317,"threads_started":1}
I20260925 15:23:45.809581 28765 heartbeater.cc:347] Connected to a master server at 127.8.226.126:46683
I20260925 15:23:45.809885 28765 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:45.810561 28765 heartbeater.cc:511] Master 127.8.226.126:46683 requested a full tablet report, sending...
I20260925 15:23:45.812029 28442 ts_manager.cc:194] Registered new tserver with Master: b39fa0fcdcee42cdb4477d31322aaedf (127.8.226.66:41549)
I20260925 15:23:45.812808 28442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.66@KRBTEST.COM'} at 127.8.226.66:50257
WARNING: no policy specified for kudu/127.8.226.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
Entry for principal kudu/127.8.226.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab.
I20260925 15:23:45.844870 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46683
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:46.160794 28770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:46.161116 28770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:46.161281 28770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:46.168488 28770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:46.168656 28770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:46.168834 28770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:46.178802 28770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.67.keytab
--principal=kudu/127.8.226.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:46.180085 28770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:46.181432 28770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:46.194882 28775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:46.194916 28778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:46.194948 28776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:46.195636 28770 server_base.cc:1034] running on GCE node
I20260925 15:23:46.196281 28770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:46.197397 28770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:46.198596 28770 hybrid_clock.cc:648] HybridClock initialized: now 1790349826198526 us; error 64 us; skew 500 ppm
I20260925 15:23:46.198935 28770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:46 dist-test-slave-8vw7 krb5kdc[28397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:46.202128 28770 init.cc:377] Logged in from keytab as kudu/127.8.226.67@KRBTEST.COM (short username kudu)
I20260925 15:23:46.203521 28770 webserver.cc:460] Webserver started at http://127.8.226.67:36539/ using document root <none> and password file <none>
I20260925 15:23:46.204206 28770 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:46.204342 28770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:46.204646 28770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:46.207572 28770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "8d97c4246648438b9a40ec1633ef035e"
format_stamp: "Formatted at 2026-09-25 15:23:46 on dist-test-slave-8vw7"
server_key: "2707a09ab23001a9d0ff3f7fefae0ae1"
server_key_iv: "0fb31ed93a0f733bc5a0ae707d3e8c32"
server_key_version: "encryptionkey@0"
I20260925 15:23:46.208371 28770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "8d97c4246648438b9a40ec1633ef035e"
format_stamp: "Formatted at 2026-09-25 15:23:46 on dist-test-slave-8vw7"
server_key: "2707a09ab23001a9d0ff3f7fefae0ae1"
server_key_iv: "0fb31ed93a0f733bc5a0ae707d3e8c32"
server_key_version: "encryptionkey@0"
I20260925 15:23:46.213470 28770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:23:46.217047 28785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:46.218336 28770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:23:46.218495 28770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "8d97c4246648438b9a40ec1633ef035e"
format_stamp: "Formatted at 2026-09-25 15:23:46 on dist-test-slave-8vw7"
server_key: "2707a09ab23001a9d0ff3f7fefae0ae1"
server_key_iv: "0fb31ed93a0f733bc5a0ae707d3e8c32"
server_key_version: "encryptionkey@0"
I20260925 15:23:46.218701 28770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:46.245267 28770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:46.249413 28770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:46.249712 28770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:46.250633 28770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:46.251916 28770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:46.252017 28770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:46.252151 28770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:46.252221 28770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:46.284452 28770 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:41449
I20260925 15:23:46.284504 28897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:41449 every 8 connection(s)
I20260925 15:23:46.285732 28770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:46.288199 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28770
I20260925 15:23:46.288393 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:46.288827 9097 external_mini_cluster.cc:1428] Setting key 0d2d8ab0981a2b83fad51555c58420cb
Sep 25 15:23:46 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.67@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:23:46.305477 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.288281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:46503 (local address 127.8.226.126:46683)
0925 15:23:46.288477 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.288483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.289759 (+ 1276us) server_negotiation.cc:407] Connection header received
0925 15:23:46.292038 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.292042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.292129 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.292223 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:46.293728 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.294486 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.296021 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.296428 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.296586 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.299889 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:46.299893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:46.299897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:46.299928 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:46.302507 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.303211 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.303218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.303224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.303362 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.303806 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.303811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.303814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.304087 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:46.304245 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.304898 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.305138 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:23:46.306412 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.288767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:46683 (local address 127.8.226.67:46503)
0925 15:23:46.289560 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:46.289631 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:46.291798 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:46.292411 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:46.292418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:46.293011 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:46.293569 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.293586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.294665 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.294675 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.295873 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.295889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.296642 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.296652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.296897 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.297626 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:46.297648 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:46.299706 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:46.302654 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:46.302661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:46.302672 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:46.303080 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:46.303491 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:46.303494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:46.303497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:46.303670 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:46.304223 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:46.304231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:46.304531 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:23:46.304789 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 15:23:46.305316 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":339,"threads_started":1}
I20260925 15:23:46.307734 28898 heartbeater.cc:347] Connected to a master server at 127.8.226.126:46683
I20260925 15:23:46.307997 28898 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:46.308672 28898 heartbeater.cc:511] Master 127.8.226.126:46683 requested a full tablet report, sending...
I20260925 15:23:46.310097 28442 ts_manager.cc:194] Registered new tserver with Master: 8d97c4246648438b9a40ec1633ef035e (127.8.226.67:41449)
I20260925 15:23:46.310842 28442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.226.67@KRBTEST.COM'} at 127.8.226.67:46503
I20260925 15:23:46.316299 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:46.323926 28631 heartbeater.cc:503] Master 127.8.226.126:46683 was elected leader, sending a full tablet report...
I20260925 15:23:46.333209 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.320494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.8.226.126:46683)
0925 15:23:46.320699 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.320707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.321193 (+ 486us) server_negotiation.cc:407] Connection header received
0925 15:23:46.321338 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.321341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.321412 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.321509 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:46.323082 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.323965 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.325445 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.325939 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.326035 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.328172 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:46.328176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:46.328179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:46.328209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:46.330822 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.331383 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.331387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.331389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.331506 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.331831 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.331834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.331836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.332140 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:46.332316 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.332665 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.332905 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:23:46.337116 28442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:46.343778 28442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:46.366689 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.355462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:44445 (local address 127.0.0.1:49100)
0925 15:23:46.357342 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:46.357394 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:46.357571 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:46.358889 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:46.358896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:46.358927 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:46.359359 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.359372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.361470 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.361478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.363747 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.363758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.365678 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.365685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.365877 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.365923 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:23:46.366081 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 15:23:46.366295 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"thread_start_us":1414,"threads_started":1}
I20260925 15:23:46.367316 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.356014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.8.226.67:41449)
0925 15:23:46.358191 (+ 2177us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.358202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.359465 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 15:23:46.359566 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.359575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.359828 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.360014 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:46.360721 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.362217 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.364665 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.365677 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.365798 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.366007 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.366791 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.366948 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2009,"thread_start_us":1964,"threads_started":1}
I20260925 15:23:46.367612 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.355584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.8.226.65:44445)
0925 15:23:46.358197 (+ 2613us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.358209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.358245 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:23:46.358355 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.358364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.358618 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.358880 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:46.359633 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.361340 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.363904 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.365459 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.365596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.365819 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.366146 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.366313 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2384,"thread_start_us":2206,"threads_started":1}
I20260925 15:23:46.368522 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.355467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:41449 (local address 127.0.0.1:40526)
0925 15:23:46.358509 (+ 3042us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:46.358522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:46.358666 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:46.359999 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:46.360003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:46.360028 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:46.360497 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.360509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.362616 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.362621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.364517 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.364529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.366153 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.366159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.366307 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.366338 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:23:46.366432 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:23:46.366548 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2955,"mutex_wait_us":234,"thread_start_us":2581,"threads_started":1}
I20260925 15:23:46.368737 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.358680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41549 (local address 127.0.0.1:38622)
0925 15:23:46.360565 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:46.360581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:46.360688 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:46.361714 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:46.361719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:46.361744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:46.362159 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.362172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.364079 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.364084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.366116 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.366134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.367929 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.367935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.368117 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.368148 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:23:46.368242 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:23:46.368364 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1744,"thread_start_us":1188,"threads_started":1}
I20260925 15:23:46.368846 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.358903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.8.226.66:41549)
0925 15:23:46.361182 (+ 2279us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.361193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.361219 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:23:46.361321 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.361328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.361518 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.361699 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:46.362335 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.363945 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.366300 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.367750 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.367888 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.368067 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.368319 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.368485 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2127,"thread_start_us":1980,"threads_started":1}
I20260925 15:23:46.371460 28566 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee546bb6f3142689764e91cfaf9ac88 (DEFAULT_TABLE table=test-table [id=7d79f4d4d9bc4f9c8bd81ec9a729aff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:46.372308 28833 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee546bb6f3142689764e91cfaf9ac88 (DEFAULT_TABLE table=test-table [id=7d79f4d4d9bc4f9c8bd81ec9a729aff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:46.372376 28700 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee546bb6f3142689764e91cfaf9ac88 (DEFAULT_TABLE table=test-table [id=7d79f4d4d9bc4f9c8bd81ec9a729aff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:46.373080 28566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee546bb6f3142689764e91cfaf9ac88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:46.373560 28833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee546bb6f3142689764e91cfaf9ac88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:46.373730 28700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee546bb6f3142689764e91cfaf9ac88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:46.384127 28916 tablet_bootstrap.cc:492] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Bootstrap starting.
I20260925 15:23:46.387275 28917 tablet_bootstrap.cc:492] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Bootstrap starting.
I20260925 15:23:46.388006 28918 tablet_bootstrap.cc:492] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Bootstrap starting.
I20260925 15:23:46.389132 28916 tablet_bootstrap.cc:654] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:46.390444 28916 log.cc:824] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:46.391462 28918 tablet_bootstrap.cc:654] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:46.391753 28917 tablet_bootstrap.cc:654] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:46.392613 28918 log.cc:824] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:46.392693 28917 log.cc:824] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:46.393992 28916 tablet_bootstrap.cc:492] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: No bootstrap required, opened a new log
I20260925 15:23:46.394359 28916 ts_tablet_manager.cc:1397] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 15:23:46.399289 28918 tablet_bootstrap.cc:492] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: No bootstrap required, opened a new log
I20260925 15:23:46.399525 28917 tablet_bootstrap.cc:492] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: No bootstrap required, opened a new log
I20260925 15:23:46.399549 28918 ts_tablet_manager.cc:1397] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 15:23:46.399778 28917 ts_tablet_manager.cc:1397] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260925 15:23:46.405365 28916 raft_consensus.cc:348] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.405809 28916 raft_consensus.cc:374] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:46.405943 28916 raft_consensus.cc:729] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97c4246648438b9a40ec1633ef035e, State: Initialized, Role: FOLLOWER
I20260925 15:23:46.406594 28916 consensus_queue.cc:260] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.407061 28918 raft_consensus.cc:348] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.407545 28918 raft_consensus.cc:374] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:46.407701 28898 heartbeater.cc:503] Master 127.8.226.126:46683 was elected leader, sending a full tablet report...
I20260925 15:23:46.407694 28918 raft_consensus.cc:729] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39fa0fcdcee42cdb4477d31322aaedf, State: Initialized, Role: FOLLOWER
I20260925 15:23:46.408334 28916 ts_tablet_manager.cc:1428] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260925 15:23:46.408356 28918 consensus_queue.cc:260] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.409418 28765 heartbeater.cc:503] Master 127.8.226.126:46683 was elected leader, sending a full tablet report...
I20260925 15:23:46.410073 28918 ts_tablet_manager.cc:1428] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 15:23:46.410418 28917 raft_consensus.cc:348] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.410825 28917 raft_consensus.cc:374] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:46.410929 28917 raft_consensus.cc:729] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058a0d9ab5604f5d9495a490b2816efd, State: Initialized, Role: FOLLOWER
I20260925 15:23:46.411476 28917 consensus_queue.cc:260] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.413830 28917 ts_tablet_manager.cc:1428] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260925 15:23:46.536824 28766 tablet.cc:2369] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:23:46.537322 28899 tablet.cc:2369] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:23:46.547024 28632 tablet.cc:2369] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:46.586352 28922 raft_consensus.cc:482] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:46.586617 28922 raft_consensus.cc:504] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.588199 28922 leader_election.cc:288] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058a0d9ab5604f5d9495a490b2816efd (127.8.226.65:44445), b39fa0fcdcee42cdb4477d31322aaedf (127.8.226.66:41549)
I20260925 15:23:46.595731 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.589164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:52243 (local address 127.8.226.65:44445)
0925 15:23:46.589401 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.589406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.589425 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:23:46.589496 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.589504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.589650 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.589760 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:46.590224 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.591253 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.593669 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.594913 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.595083 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.595158 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.595376 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.595494 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:23:46.595782 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.588463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:44445 (local address 127.8.226.67:52243)
0925 15:23:46.588639 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:46.588662 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:46.588791 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:46.589792 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:46.589797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:46.589865 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:46.590056 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.590064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.591381 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.591384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.593490 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.593504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.595058 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.595062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.595185 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.595216 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:23:46.595334 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 15:23:46.595472 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":68,"spinlock_wait_cycles":21248}
I20260925 15:23:46.596055 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.588862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:40675 (local address 127.8.226.66:41549)
0925 15:23:46.589233 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.589240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.589795 (+ 555us) server_negotiation.cc:407] Connection header received
0925 15:23:46.589974 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.589978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.590035 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.590108 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:46.590626 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.591552 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.593897 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.595156 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.595327 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.595396 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.595684 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.595829 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260925 15:23:46.596195 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.588465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:41549 (local address 127.8.226.67:40675)
0925 15:23:46.589682 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:46.589695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:46.589831 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:46.590150 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:46.590157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:46.590183 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:46.590467 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.590474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.591686 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.591691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.593749 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:46.593759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.595359 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.595365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:46.595514 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.595544 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:23:46.595634 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:23:46.595746 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":223,"thread_start_us":634,"threads_started":1}
I20260925 15:23:46.597004 28586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee546bb6f3142689764e91cfaf9ac88" candidate_uuid: "8d97c4246648438b9a40ec1633ef035e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058a0d9ab5604f5d9495a490b2816efd" is_pre_election: true
I20260925 15:23:46.597282 28720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee546bb6f3142689764e91cfaf9ac88" candidate_uuid: "8d97c4246648438b9a40ec1633ef035e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" is_pre_election: true
I20260925 15:23:46.597486 28586 raft_consensus.cc:2454] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97c4246648438b9a40ec1633ef035e in term 0.
I20260925 15:23:46.597831 28720 raft_consensus.cc:2454] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97c4246648438b9a40ec1633ef035e in term 0.
I20260925 15:23:46.598258 28787 leader_election.cc:302] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058a0d9ab5604f5d9495a490b2816efd, 8d97c4246648438b9a40ec1633ef035e; no voters:
I20260925 15:23:46.598656 28922 raft_consensus.cc:2781] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:46.598810 28922 raft_consensus.cc:482] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:46.598928 28922 raft_consensus.cc:3037] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:46.600922 28922 raft_consensus.cc:504] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.601548 28922 leader_election.cc:288] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [CANDIDATE]: Term 1 election: Requested vote from peers 058a0d9ab5604f5d9495a490b2816efd (127.8.226.65:44445), b39fa0fcdcee42cdb4477d31322aaedf (127.8.226.66:41549)
I20260925 15:23:46.601876 28586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee546bb6f3142689764e91cfaf9ac88" candidate_uuid: "8d97c4246648438b9a40ec1633ef035e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058a0d9ab5604f5d9495a490b2816efd"
I20260925 15:23:46.602052 28586 raft_consensus.cc:3037] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:46.602037 28720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee546bb6f3142689764e91cfaf9ac88" candidate_uuid: "8d97c4246648438b9a40ec1633ef035e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39fa0fcdcee42cdb4477d31322aaedf"
I20260925 15:23:46.602222 28720 raft_consensus.cc:3037] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:46.604257 28586 raft_consensus.cc:2454] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97c4246648438b9a40ec1633ef035e in term 1.
I20260925 15:23:46.604282 28720 raft_consensus.cc:2454] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97c4246648438b9a40ec1633ef035e in term 1.
I20260925 15:23:46.604669 28788 leader_election.cc:302] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d97c4246648438b9a40ec1633ef035e, b39fa0fcdcee42cdb4477d31322aaedf; no voters:
I20260925 15:23:46.605000 28922 raft_consensus.cc:2781] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:46.605433 28922 raft_consensus.cc:686] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 1 LEADER]: Becoming Leader. State: Replica: 8d97c4246648438b9a40ec1633ef035e, State: Running, Role: LEADER
I20260925 15:23:46.605876 28922 consensus_queue.cc:237] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } }
I20260925 15:23:46.610440 28442 catalog_manager.cc:5495] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d97c4246648438b9a40ec1633ef035e (127.8.226.67). New cstate: current_term: 1 leader_uuid: "8d97c4246648438b9a40ec1633ef035e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d97c4246648438b9a40ec1633ef035e" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 41449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 } health_report { overall_health: UNKNOWN } } }
I20260925 15:23:46.657258 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.652332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.8.226.67:41449)
0925 15:23:46.652516 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.652520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.652539 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:23:46.652645 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.652649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.652725 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.652847 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:46.653242 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.653992 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.655313 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.655681 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.655768 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.656213 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:46.656707 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:46.656812 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.656990 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.657063 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:23:46.665166 28720 raft_consensus.cc:1261] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 1 FOLLOWER]: Refusing update from remote peer 8d97c4246648438b9a40ec1633ef035e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:46.665166 28586 raft_consensus.cc:1261] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 1 FOLLOWER]: Refusing update from remote peer 8d97c4246648438b9a40ec1633ef035e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:46.666271 28926 consensus_queue.cc:1035] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39fa0fcdcee42cdb4477d31322aaedf" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 41549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:46.666615 28922 consensus_queue.cc:1035] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [LEADER]: Connected to new peer: Peer: permanent_uuid: "058a0d9ab5604f5d9495a490b2816efd" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 44445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:46.680399 28929 mvcc.cc:205] Tried to move back new op lower bound from 7333272890010255360 to 7333272889781153792. Current Snapshot: MvccSnapshot[applied={T|T < 7333272890010255360}]
I20260925 15:23:46.684021 28930 mvcc.cc:205] Tried to move back new op lower bound from 7333272890010255360 to 7333272889781153792. Current Snapshot: MvccSnapshot[applied={T|T < 7333272890010255360}]
I20260925 15:23:46.695261 28442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:46.695518 28442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:46.698907 28442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e433eb94fe8d42d4aa44b00d39444503: Sending DeleteTablet for 3 replicas of tablet 3ee546bb6f3142689764e91cfaf9ac88
I20260925 15:23:46.699961 28566 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee546bb6f3142689764e91cfaf9ac88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:46 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:49100
I20260925 15:23:46.700114 28700 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee546bb6f3142689764e91cfaf9ac88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:46 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:38622
I20260925 15:23:46.700232 28833 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee546bb6f3142689764e91cfaf9ac88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:46 UTC) from {username='kudu', principal='kudu/127.8.226.126@KRBTEST.COM'} at 127.0.0.1:40526
I20260925 15:23:46.700871 28939 tablet_replica.cc:330] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: stopping tablet replica
I20260925 15:23:46.701309 28939 raft_consensus.cc:2229] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:46.701861 28939 raft_consensus.cc:2258] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:46.701886 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28503
I20260925 15:23:46.708410 28939 ts_tablet_manager.cc:1905] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:46.721477 28940 tablet_replica.cc:330] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: stopping tablet replica
I20260925 15:23:46.722031 28940 raft_consensus.cc:2229] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:46.722600 28940 raft_consensus.cc:2258] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:46.723398 28941 tablet_replica.cc:330] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: stopping tablet replica
I20260925 15:23:46.723773 28941 raft_consensus.cc:2229] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:46.724434 28941 raft_consensus.cc:2258] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:46.724654 28940 ts_tablet_manager.cc:1905] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:46.726521 28941 ts_tablet_manager.cc:1905] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:46.728885 28939 ts_tablet_manager.cc:1918] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:46.729120 28939 log.cc:1196] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-0/wal/wals/3ee546bb6f3142689764e91cfaf9ac88
I20260925 15:23:46.729697 28939 ts_tablet_manager.cc:1939] T 3ee546bb6f3142689764e91cfaf9ac88 P 058a0d9ab5604f5d9495a490b2816efd: Deleting consensus metadata
I20260925 15:23:46.731479 28940 ts_tablet_manager.cc:1918] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:46.731696 28940 log.cc:1196] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/3ee546bb6f3142689764e91cfaf9ac88
I20260925 15:23:46.732283 28940 ts_tablet_manager.cc:1939] T 3ee546bb6f3142689764e91cfaf9ac88 P b39fa0fcdcee42cdb4477d31322aaedf: Deleting consensus metadata
I20260925 15:23:46.733155 28427 catalog_manager.cc:4841] TS 058a0d9ab5604f5d9495a490b2816efd (127.8.226.65:44445): tablet 3ee546bb6f3142689764e91cfaf9ac88 (table test-table [id=7d79f4d4d9bc4f9c8bd81ec9a729aff0]) successfully deleted
I20260925 15:23:46.733960 28941 ts_tablet_manager.cc:1918] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:46.733964 28428 catalog_manager.cc:4841] TS b39fa0fcdcee42cdb4477d31322aaedf (127.8.226.66:41549): tablet 3ee546bb6f3142689764e91cfaf9ac88 (table test-table [id=7d79f4d4d9bc4f9c8bd81ec9a729aff0]) successfully deleted
I20260925 15:23:46.734149 28941 log.cc:1196] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/3ee546bb6f3142689764e91cfaf9ac88
I20260925 15:23:46.734686 28941 ts_tablet_manager.cc:1939] T 3ee546bb6f3142689764e91cfaf9ac88 P 8d97c4246648438b9a40ec1633ef035e: Deleting consensus metadata
I20260925 15:23:46.736302 28428 catalog_manager.cc:4841] TS 8d97c4246648438b9a40ec1633ef035e (127.8.226.67:41449): tablet 3ee546bb6f3142689764e91cfaf9ac88 (table test-table [id=7d79f4d4d9bc4f9c8bd81ec9a729aff0]) successfully deleted
Sep 25 15:23:46 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.65@KRBTEST.COM
I20260925 15:23:46.744925 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.702841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.8.226.65:44445)
0925 15:23:46.709105 (+ 6264us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.709113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.723013 (+ 13900us) server_negotiation.cc:407] Connection header received
0925 15:23:46.723476 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.723481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.723561 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.723636 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:46.725619 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.726732 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.729852 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.730459 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.730555 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.737927 (+ 7372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:46.737944 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:46.737963 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:46.738008 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:46.741912 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.742676 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.742684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.742688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.742839 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.743258 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.743263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.743266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.743708 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:46.744003 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.744295 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.744554 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6129}
I20260925 15:23:46.745728 28626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:46.849050 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28503
I20260925 15:23:46.866950 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28637
Sep 25 15:23:46 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.66@KRBTEST.COM
I20260925 15:23:46.883812 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.867750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.8.226.66:41549)
0925 15:23:46.867950 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.867957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.867981 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:23:46.868060 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.868065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.868174 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.868293 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:46.869707 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.870839 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.872240 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.872829 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:46.872972 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:46.877161 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:46.877179 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:46.877199 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:46.877253 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:46.880793 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.881716 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.881725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.881729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.881874 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:46.882197 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:46.882204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:46.882214 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:46.882681 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:46.882993 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:46.883229 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:23:46.883466 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:23:46.884436 28760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:46.972393 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28637
I20260925 15:23:46.994453 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28770
Sep 25 15:23:47 dist-test-slave-8vw7 krb5kdc[28397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.67@KRBTEST.COM
I20260925 15:23:47.012239 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:46.995342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.8.226.67:41449)
0925 15:23:46.995530 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:46.995537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:46.995559 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:23:46.995768 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:46.995772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:46.995830 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:46.995922 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:46.997737 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:46.998483 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:47.000380 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:47.000795 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:47.000879 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:47.006252 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:47.006264 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:47.006278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:47.006308 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:47.009470 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:47.010142 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:47.010149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:47.010151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:47.010301 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:47.010674 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:47.010684 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:47.010687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:47.011042 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:47.011346 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:47.011640 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:23:47.011867 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:23:47.012887 28893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:47.135355 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28770
I20260925 15:23:47.155704 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28410
I20260925 15:23:47.156836 28472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:47.263759 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28410
2026-09-25T15:23:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4320 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:47Z Disabled control of system clock
I20260925 15:23:47.336136 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35513
--webserver_interface=127.8.226.126
--webserver_port=0
--builtin_ntp_servers=127.8.226.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:23:47.649621 28950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:47.649869 28950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:47.649945 28950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:47.656905 28950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:47.657017 28950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:47.657056 28950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:47.657119 28950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:47.657166 28950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:47.665956 28950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:35513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:47.667071 28950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:47.668756 28950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:47.679853 28955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:47.679945 28956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:47.679885 28958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:47.680913 28950 server_base.cc:1034] running on GCE node
I20260925 15:23:47.681651 28950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:47.682880 28950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:47.684085 28950 hybrid_clock.cc:648] HybridClock initialized: now 1790349827684010 us; error 53 us; skew 500 ppm
I20260925 15:23:47.684415 28950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:47.686501 28950 webserver.cc:460] Webserver started at http://127.8.226.126:35221/ using document root <none> and password file <none>
I20260925 15:23:47.687127 28950 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:47.687234 28950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:47.687544 28950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:47.690268 28950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "5fc6f9582c6b46e68159185dc0a9dca3"
format_stamp: "Formatted at 2026-09-25 15:23:47 on dist-test-slave-8vw7"
server_key: "a891ee8cb0b4a937d33dac6b8ba86a4d"
server_key_iv: "afdba667911ad4c5a25b91a44ddf2aab"
server_key_version: "encryptionkey@0"
I20260925 15:23:47.691018 28950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "5fc6f9582c6b46e68159185dc0a9dca3"
format_stamp: "Formatted at 2026-09-25 15:23:47 on dist-test-slave-8vw7"
server_key: "a891ee8cb0b4a937d33dac6b8ba86a4d"
server_key_iv: "afdba667911ad4c5a25b91a44ddf2aab"
server_key_version: "encryptionkey@0"
I20260925 15:23:47.695864 28950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 15:23:47.699023 28964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:47.700297 28950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:23:47.700467 28950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "5fc6f9582c6b46e68159185dc0a9dca3"
format_stamp: "Formatted at 2026-09-25 15:23:47 on dist-test-slave-8vw7"
server_key: "a891ee8cb0b4a937d33dac6b8ba86a4d"
server_key_iv: "afdba667911ad4c5a25b91a44ddf2aab"
server_key_version: "encryptionkey@0"
I20260925 15:23:47.700657 28950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:47.718322 28950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:47.719058 28950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:47.719332 28950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:47.737138 29015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:35513 every 8 connection(s)
I20260925 15:23:47.737157 28950 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:35513
I20260925 15:23:47.738561 28950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:47.742046 29016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:47.745785 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 28950
I20260925 15:23:47.745987 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:47.746527 9097 external_mini_cluster.cc:1428] Setting key 82bbc4a69a9e831df9178641a1824067
I20260925 15:23:47.754578 29016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3: Bootstrap starting.
I20260925 15:23:47.758064 29016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:47.759131 29016 log.cc:824] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:47.759603 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:47.748371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.8.226.126:35513)
0925 15:23:47.749327 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:47.749336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:47.749374 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:23:47.750909 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:47.750929 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:47.751372 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:47.751677 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:47.752279 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:47.753553 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:47.755383 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:47.756015 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:47.756136 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:47.757354 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:47.757361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:47.757372 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:47.757877 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:47.757959 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:47.757978 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:47.758067 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:47.758317 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:23:47.758613 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":356,"threads_started":1}
I20260925 15:23:47.761909 29016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3: No bootstrap required, opened a new log
I20260925 15:23:47.768718 29016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35513 } }
I20260925 15:23:47.769052 29016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:47.769143 29016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc6f9582c6b46e68159185dc0a9dca3, State: Initialized, Role: FOLLOWER
I20260925 15:23:47.769631 29016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35513 } }
I20260925 15:23:47.769825 29016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:47.769917 29016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:47.770025 29016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:47.771849 29016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35513 } }
I20260925 15:23:47.772233 29016 leader_election.cc:302] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc6f9582c6b46e68159185dc0a9dca3; no voters:
I20260925 15:23:47.772761 29016 leader_election.cc:288] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:47.772881 29021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:47.773725 29021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc6f9582c6b46e68159185dc0a9dca3, State: Running, Role: LEADER
I20260925 15:23:47.774189 29021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35513 } }
I20260925 15:23:47.774693 29016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:47.777160 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35513 } } }
I20260925 15:23:47.777158 29023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fc6f9582c6b46e68159185dc0a9dca3. Latest consensus state: current_term: 1 leader_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc6f9582c6b46e68159185dc0a9dca3" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 35513 } } }
I20260925 15:23:47.777613 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:47.777640 29023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:47.783720 29030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:47.788885 29030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:47.796334 29030 catalog_manager.cc:1347] Generated new cluster ID: 6849cb4b62684d40b62050e29bd37b44
I20260925 15:23:47.796445 29030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:47.813517 29030 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:47.814675 29030 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:47.819445 29030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3: Generated new TSK 0
I20260925 15:23:47.820151 29030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:47.830803 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.65:0
--local_ip_for_outbound_sockets=127.8.226.65
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35513
--builtin_ntp_servers=127.8.226.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:23:48.156747 29040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:48.157065 29040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:48.157231 29040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:48.164068 29040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:48.164202 29040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:48.164383 29040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.65
I20260925 15:23:48.173084 29040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.226.65
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.65
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:48.174335 29040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:48.176229 29040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:48.189138 29046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:48.189170 29045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:48.189221 29048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:48.189860 29040 server_base.cc:1034] running on GCE node
I20260925 15:23:48.190461 29040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:48.191306 29040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:48.192523 29040 hybrid_clock.cc:648] HybridClock initialized: now 1790349828192487 us; error 48 us; skew 500 ppm
I20260925 15:23:48.192840 29040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:48.194927 29040 webserver.cc:460] Webserver started at http://127.8.226.65:36971/ using document root <none> and password file <none>
I20260925 15:23:48.195583 29040 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:48.195708 29040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:48.196012 29040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:48.198839 29040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data/instance:
uuid: "bdab9ca973374703b4e08692e05af645"
format_stamp: "Formatted at 2026-09-25 15:23:48 on dist-test-slave-8vw7"
server_key: "7aca219cd8ad517e5f07da76d7df2065"
server_key_iv: "7ae2650c678b59ee7ba95789fff0829c"
server_key_version: "encryptionkey@0"
I20260925 15:23:48.199599 29040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance:
uuid: "bdab9ca973374703b4e08692e05af645"
format_stamp: "Formatted at 2026-09-25 15:23:48 on dist-test-slave-8vw7"
server_key: "7aca219cd8ad517e5f07da76d7df2065"
server_key_iv: "7ae2650c678b59ee7ba95789fff0829c"
server_key_version: "encryptionkey@0"
I20260925 15:23:48.204409 29040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:23:48.207751 29054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:48.209043 29040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:48.209198 29040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal
uuid: "bdab9ca973374703b4e08692e05af645"
format_stamp: "Formatted at 2026-09-25 15:23:48 on dist-test-slave-8vw7"
server_key: "7aca219cd8ad517e5f07da76d7df2065"
server_key_iv: "7ae2650c678b59ee7ba95789fff0829c"
server_key_version: "encryptionkey@0"
I20260925 15:23:48.209416 29040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:48.252203 29040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:48.253137 29040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:48.253443 29040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:48.254387 29040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:48.255810 29040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:48.255930 29040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:48.256069 29040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:48.256155 29040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:48.288182 29040 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.65:46527
I20260925 15:23:48.288232 29166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.65:46527 every 8 connection(s)
I20260925 15:23:48.289485 29040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:48.294564 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29040
I20260925 15:23:48.294732 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:48.295256 9097 external_mini_cluster.cc:1428] Setting key 50e00bb6f2877b54752df05cfdf50a4f
I20260925 15:23:48.298061 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.66:0
--local_ip_for_outbound_sockets=127.8.226.66
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35513
--builtin_ntp_servers=127.8.226.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:23:48.302412 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:48.291972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.65:57983 (local address 127.8.226.126:35513)
0925 15:23:48.292392 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:48.292397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:48.293547 (+ 1150us) server_negotiation.cc:407] Connection header received
0925 15:23:48.295914 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:48.295921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:48.296014 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:48.296126 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:48.297402 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.298105 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.299641 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.300069 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.300197 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:48.301575 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:48.301581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:48.301585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:48.301613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:48.301653 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:48.301670 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:48.301765 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:48.301991 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:23:48.302085 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":236,"threads_started":1}
I20260925 15:23:48.303393 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:48.292504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35513 (local address 127.8.226.65:57983)
0925 15:23:48.293395 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:48.293436 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:48.295602 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:48.296384 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:48.296392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:48.296480 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:48.297232 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:48.297258 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.298262 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.298267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:48.299478 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:48.299490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.300314 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.300321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:48.300544 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:48.301284 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:48.301287 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:48.301322 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:48.301393 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:48.301764 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:48.301778 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:48.301787 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:23:48.301968 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 15:23:48.302343 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260925 15:23:48.305107 29167 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35513
I20260925 15:23:48.305365 29167 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:48.305980 29167 heartbeater.cc:511] Master 127.8.226.126:35513 requested a full tablet report, sending...
I20260925 15:23:48.307811 28981 ts_manager.cc:194] Registered new tserver with Master: bdab9ca973374703b4e08692e05af645 (127.8.226.65:46527)
I20260925 15:23:48.309307 28981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.65:57983
W20260925 15:23:48.621665 29171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:48.621960 29171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:48.622123 29171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:48.629208 29171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:48.629324 29171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:48.629487 29171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.66
I20260925 15:23:48.638279 29171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.226.66
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.66
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:48.639588 29171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:48.641281 29171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:48.653903 29179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:48.653960 29176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:48.653867 29177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:48.654518 29171 server_base.cc:1034] running on GCE node
I20260925 15:23:48.655169 29171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:48.655972 29171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:48.657195 29171 hybrid_clock.cc:648] HybridClock initialized: now 1790349828657152 us; error 55 us; skew 500 ppm
I20260925 15:23:48.657496 29171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:48.659595 29171 webserver.cc:460] Webserver started at http://127.8.226.66:34645/ using document root <none> and password file <none>
I20260925 15:23:48.660271 29171 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:48.660413 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:48.660719 29171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:48.663503 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data/instance:
uuid: "d94df4f15b644171b6074292bf80afc5"
format_stamp: "Formatted at 2026-09-25 15:23:48 on dist-test-slave-8vw7"
server_key: "80c7bdf296c7a41c3ea3280722c2fa7c"
server_key_iv: "c5c1079b9866553d4c233483844023e4"
server_key_version: "encryptionkey@0"
I20260925 15:23:48.664320 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance:
uuid: "d94df4f15b644171b6074292bf80afc5"
format_stamp: "Formatted at 2026-09-25 15:23:48 on dist-test-slave-8vw7"
server_key: "80c7bdf296c7a41c3ea3280722c2fa7c"
server_key_iv: "c5c1079b9866553d4c233483844023e4"
server_key_version: "encryptionkey@0"
I20260925 15:23:48.669210 29171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:23:48.672569 29185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:48.673883 29171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:48.674067 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal
uuid: "d94df4f15b644171b6074292bf80afc5"
format_stamp: "Formatted at 2026-09-25 15:23:48 on dist-test-slave-8vw7"
server_key: "80c7bdf296c7a41c3ea3280722c2fa7c"
server_key_iv: "c5c1079b9866553d4c233483844023e4"
server_key_version: "encryptionkey@0"
I20260925 15:23:48.674296 29171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:48.690165 29171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:48.690917 29171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:48.691227 29171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:48.692113 29171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:48.693712 29171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:48.693830 29171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:48.693953 29171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:48.694041 29171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:48.725658 29171 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.66:45873
I20260925 15:23:48.725713 29297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.66:45873 every 8 connection(s)
I20260925 15:23:48.726922 29171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:48.734242 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29171
I20260925 15:23:48.734422 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:48.734933 9097 external_mini_cluster.cc:1428] Setting key aaed97d8bced8e361489022d08e8d056
I20260925 15:23:48.737998 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.226.67:0
--local_ip_for_outbound_sockets=127.8.226.67
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35513
--builtin_ntp_servers=127.8.226.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:23:48.739723 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:48.729455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:43009 (local address 127.8.226.126:35513)
0925 15:23:48.729635 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:48.729640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:48.730978 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 15:23:48.732620 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:48.732624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:48.732689 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:48.732773 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:48.733813 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.734624 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.736757 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.737279 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.737418 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:48.738812 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:48.738818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:48.738823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:48.738850 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:48.738894 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:48.738905 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:48.738998 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:48.739347 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 15:23:48.739434 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:23:48.740695 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:48.729969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35513 (local address 127.8.226.66:43009)
0925 15:23:48.730826 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:48.730866 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:48.732394 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:48.733015 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:48.733028 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:48.733112 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:48.733649 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:48.733665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.734769 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.734776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:48.736622 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:48.736635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:48.737522 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:48.737530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:48.737756 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:48.738495 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:48.738498 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:48.738535 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:48.738602 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:48.738996 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:48.739008 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:48.739017 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:23:48.739300 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 15:23:48.739661 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":269,"threads_started":1}
I20260925 15:23:48.742154 29298 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35513
I20260925 15:23:48.742453 29298 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:48.743146 29298 heartbeater.cc:511] Master 127.8.226.126:35513 requested a full tablet report, sending...
I20260925 15:23:48.744469 28981 ts_manager.cc:194] Registered new tserver with Master: d94df4f15b644171b6074292bf80afc5 (127.8.226.66:45873)
I20260925 15:23:48.745375 28981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.66:43009
W20260925 15:23:49.074218 29301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:49.074481 29301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:49.074621 29301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:49.082345 29301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:49.082441 29301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:49.082548 29301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.226.67
I20260925 15:23:49.091825 29301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.226.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.226.67
--webserver_port=0
--tserver_master_addrs=127.8.226.126:35513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.226.67
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:49.093063 29301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:49.094745 29301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:49.107187 29309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:49.107286 29307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:49.107322 29306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:49.108134 29301 server_base.cc:1034] running on GCE node
I20260925 15:23:49.108666 29301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:49.109663 29301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:49.110885 29301 hybrid_clock.cc:648] HybridClock initialized: now 1790349829110848 us; error 45 us; skew 500 ppm
I20260925 15:23:49.111207 29301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:23:49.113407 29301 webserver.cc:460] Webserver started at http://127.8.226.67:34011/ using document root <none> and password file <none>
I20260925 15:23:49.114086 29301 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:49.114233 29301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:49.114557 29301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:49.117316 29301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data/instance:
uuid: "9f63d2aef76c4d5494a15dfb09d9cd13"
format_stamp: "Formatted at 2026-09-25 15:23:49 on dist-test-slave-8vw7"
server_key: "d432417e5f23feefa31d3aeee09bcf82"
server_key_iv: "e450bb55896b70cc6f0b5f4d340eb57c"
server_key_version: "encryptionkey@0"
I20260925 15:23:49.118120 29301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance:
uuid: "9f63d2aef76c4d5494a15dfb09d9cd13"
format_stamp: "Formatted at 2026-09-25 15:23:49 on dist-test-slave-8vw7"
server_key: "d432417e5f23feefa31d3aeee09bcf82"
server_key_iv: "e450bb55896b70cc6f0b5f4d340eb57c"
server_key_version: "encryptionkey@0"
I20260925 15:23:49.122727 29301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:23:49.125963 29315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:49.127148 29301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:23:49.127307 29301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal
uuid: "9f63d2aef76c4d5494a15dfb09d9cd13"
format_stamp: "Formatted at 2026-09-25 15:23:49 on dist-test-slave-8vw7"
server_key: "d432417e5f23feefa31d3aeee09bcf82"
server_key_iv: "e450bb55896b70cc6f0b5f4d340eb57c"
server_key_version: "encryptionkey@0"
I20260925 15:23:49.127512 29301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:49.150187 29301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:49.151060 29301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:49.151381 29301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:49.152354 29301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:49.153882 29301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:49.154000 29301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:49.154142 29301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:49.154230 29301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:49.186952 29301 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.67:34075
I20260925 15:23:49.187032 29427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.67:34075 every 8 connection(s)
I20260925 15:23:49.188215 29301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:49.190246 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29301
I20260925 15:23:49.190415 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:49.190969 9097 external_mini_cluster.cc:1428] Setting key fe186b547509d4c5893710c4cab1e5a8
I20260925 15:23:49.200450 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.190858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.67:41121 (local address 127.8.226.126:35513)
0925 15:23:49.191085 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.191091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.192688 (+ 1597us) server_negotiation.cc:407] Connection header received
0925 15:23:49.194763 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.194767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.194831 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.194920 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:49.195877 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.196642 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.198157 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.198576 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.198674 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.199834 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:49.199839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:49.199842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:49.199859 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:49.199889 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:49.199900 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:49.199970 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.200158 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.200226 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:23:49.201418 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.191434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:35513 (local address 127.8.226.67:41121)
0925 15:23:49.192516 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.192572 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:49.194515 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:49.195101 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:49.195108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:49.195206 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:49.195735 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.195750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.196770 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.196774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.198027 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.198038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.198742 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.198746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.198933 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.199556 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:23:49.199559 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:49.199594 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:23:49.199665 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:49.199981 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:49.199988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:49.199992 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:23:49.200130 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:23:49.200403 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":363,"threads_started":1}
I20260925 15:23:49.202711 29428 heartbeater.cc:347] Connected to a master server at 127.8.226.126:35513
I20260925 15:23:49.203025 29428 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:49.203733 29428 heartbeater.cc:511] Master 127.8.226.126:35513 requested a full tablet report, sending...
I20260925 15:23:49.205255 28981 ts_manager.cc:194] Registered new tserver with Master: 9f63d2aef76c4d5494a15dfb09d9cd13 (127.8.226.67:34075)
I20260925 15:23:49.206249 28981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.226.67:41121
I20260925 15:23:49.206315 9097 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:49.216404 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.210856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.8.226.126:35513)
0925 15:23:49.210993 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.210997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.211422 (+ 425us) server_negotiation.cc:407] Connection header received
0925 15:23:49.211607 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.211612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.211694 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.211791 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:49.212246 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.213003 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.214325 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.214732 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.214861 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.215764 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:49.215770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:49.215774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:49.215799 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:49.215827 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:49.215838 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:49.215925 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.216057 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.216137 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:23:49.219702 28981 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:49.226469 28981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:49.265265 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.238889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.8.226.66:45873)
0925 15:23:49.255883 (+ 16994us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.255894 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.257022 (+ 1128us) server_negotiation.cc:407] Connection header received
0925 15:23:49.257197 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.257204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.257440 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.257632 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:49.258663 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.260152 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.262718 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.263952 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.264082 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.264300 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.264743 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.264913 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16806,"thread_start_us":16673,"threads_started":1}
I20260925 15:23:49.265904 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.237881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41446)
0925 15:23:49.254223 (+ 16342us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.254287 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:49.254518 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:49.258449 (+ 3931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:49.258454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:49.258478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:49.258842 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.258856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.261029 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.261033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.263295 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.263311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.265071 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.265077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.265273 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.265315 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:23:49.265387 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:23:49.265501 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15983,"thread_start_us":15083,"threads_started":1}
I20260925 15:23:49.266043 29443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.238246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:34075 (local address 127.0.0.1:42118)
0925 15:23:49.256447 (+ 18201us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.256469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:49.256737 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:49.257872 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:49.257879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:49.257929 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:49.258308 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.258317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.260311 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.260317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.263020 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.263033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.265256 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.265261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.265453 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.265478 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:23:49.265550 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:23:49.265645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18038,"mutex_wait_us":124,"spinlock_wait_cycles":21760,"thread_start_us":18262,"threads_started":1}
I20260925 15:23:49.266093 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.238798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.8.226.67:34075)
0925 15:23:49.255346 (+ 16548us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.255359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.256587 (+ 1228us) server_negotiation.cc:407] Connection header received
0925 15:23:49.256996 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.257003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.257319 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.257495 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:49.258590 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.260145 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.263258 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.264696 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.264823 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.265126 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.265614 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.265776 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16314,"thread_start_us":16135,"threads_started":1}
I20260925 15:23:49.266389 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.238070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.8.226.65:46527)
0925 15:23:49.257910 (+ 19840us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.257918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.257943 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:23:49.258041 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.258050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.258251 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.258446 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:49.259034 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.260858 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.263484 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.264839 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.265602 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.265820 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.265887 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.266043 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19697,"thread_start_us":19580,"threads_started":1}
I20260925 15:23:49.265304 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.238323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.66:45873 (local address 127.0.0.1:37650)
0925 15:23:49.256767 (+ 18444us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.256784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:49.256907 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:49.257872 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:49.257879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:49.257908 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:49.258446 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.258460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.260320 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.260324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.262494 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.262505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.264395 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.264399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.264521 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.264552 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:23:49.264682 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:23:49.264888 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18331,"mutex_wait_us":141,"thread_start_us":17791,"threads_started":1}
I20260925 15:23:49.269595 29363 tablet_service.cc:1467] Processing CreateTablet for tablet b447a9e3ff78451eac2ed6b7392c0012 (DEFAULT_TABLE table=test-table [id=f8aed80b5ca8490688e8f9bb95a08b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:49.269580 29102 tablet_service.cc:1467] Processing CreateTablet for tablet b447a9e3ff78451eac2ed6b7392c0012 (DEFAULT_TABLE table=test-table [id=f8aed80b5ca8490688e8f9bb95a08b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:49.270802 29363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b447a9e3ff78451eac2ed6b7392c0012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:49.270884 29102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b447a9e3ff78451eac2ed6b7392c0012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:49.272154 29233 tablet_service.cc:1467] Processing CreateTablet for tablet b447a9e3ff78451eac2ed6b7392c0012 (DEFAULT_TABLE table=test-table [id=f8aed80b5ca8490688e8f9bb95a08b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:49.273392 29233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b447a9e3ff78451eac2ed6b7392c0012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:49.284267 29446 tablet_bootstrap.cc:492] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Bootstrap starting.
I20260925 15:23:49.286784 29447 tablet_bootstrap.cc:492] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: Bootstrap starting.
I20260925 15:23:49.288508 29446 tablet_bootstrap.cc:654] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:49.289757 29446 log.cc:824] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:49.290565 29447 tablet_bootstrap.cc:654] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:49.292003 29447 log.cc:824] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:49.293195 29448 tablet_bootstrap.cc:492] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Bootstrap starting.
I20260925 15:23:49.296509 29448 tablet_bootstrap.cc:654] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:49.299227 29448 log.cc:824] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:49.302068 29446 tablet_bootstrap.cc:492] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: No bootstrap required, opened a new log
I20260925 15:23:49.302424 29446 ts_tablet_manager.cc:1397] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260925 15:23:49.306463 29448 tablet_bootstrap.cc:492] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: No bootstrap required, opened a new log
I20260925 15:23:49.306710 29447 tablet_bootstrap.cc:492] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: No bootstrap required, opened a new log
I20260925 15:23:49.306730 29448 ts_tablet_manager.cc:1397] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260925 15:23:49.306955 29447 ts_tablet_manager.cc:1397] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20260925 15:23:49.312722 29167 heartbeater.cc:503] Master 127.8.226.126:35513 was elected leader, sending a full tablet report...
I20260925 15:23:49.313187 29446 raft_consensus.cc:348] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.313627 29446 raft_consensus.cc:374] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:49.313709 29446 raft_consensus.cc:729] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f63d2aef76c4d5494a15dfb09d9cd13, State: Initialized, Role: FOLLOWER
I20260925 15:23:49.314397 29446 consensus_queue.cc:260] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.314436 29447 raft_consensus.cc:348] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.314888 29447 raft_consensus.cc:374] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:49.315013 29447 raft_consensus.cc:729] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdab9ca973374703b4e08692e05af645, State: Initialized, Role: FOLLOWER
I20260925 15:23:49.315919 29447 consensus_queue.cc:260] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.317536 29448 raft_consensus.cc:348] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.318121 29448 raft_consensus.cc:374] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:49.318295 29448 raft_consensus.cc:729] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94df4f15b644171b6074292bf80afc5, State: Initialized, Role: FOLLOWER
I20260925 15:23:49.319159 29448 consensus_queue.cc:260] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.320156 29447 ts_tablet_manager.cc:1428] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260925 15:23:49.321133 29428 heartbeater.cc:503] Master 127.8.226.126:35513 was elected leader, sending a full tablet report...
I20260925 15:23:49.321969 29446 ts_tablet_manager.cc:1428] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Time spent starting tablet: real 0.019s user 0.013s sys 0.007s
I20260925 15:23:49.322817 29298 heartbeater.cc:503] Master 127.8.226.126:35513 was elected leader, sending a full tablet report...
I20260925 15:23:49.323882 29448 ts_tablet_manager.cc:1428] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260925 15:23:49.370400 29454 raft_consensus.cc:482] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:49.370723 29454 raft_consensus.cc:504] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.372466 29454 leader_election.cc:288] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdab9ca973374703b4e08692e05af645 (127.8.226.65:46527), 9f63d2aef76c4d5494a15dfb09d9cd13 (127.8.226.67:34075)
I20260925 15:23:49.380200 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.373090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:51733 (local address 127.8.226.67:34075)
0925 15:23:49.373323 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.373332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.374341 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 15:23:49.374525 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.374530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.374622 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.374726 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:49.375420 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.376250 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.378497 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.379313 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.379515 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.379588 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.379851 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.379969 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:23:49.380353 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.372933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.67:34075 (local address 127.8.226.66:51733)
0925 15:23:49.374226 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.374245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:49.374385 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:49.374725 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:49.374730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:49.374757 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:49.375240 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.375249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.376376 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.376382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.378184 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.378197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.379505 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.379511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.379675 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.379717 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:23:49.379824 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:23:49.379951 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":67,"spinlock_wait_cycles":21376,"thread_start_us":727,"threads_started":1}
I20260925 15:23:49.380466 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.372946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.226.66:53027 (local address 127.8.226.65:46527)
0925 15:23:49.373259 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.373268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.374249 (+ 981us) server_negotiation.cc:407] Connection header received
0925 15:23:49.374466 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.374472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.374551 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.374649 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:49.375414 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.376195 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.378557 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.379566 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.379758 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.379822 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.380095 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.380215 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 15:23:49.380610 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.372907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.8.226.66:53027)
0925 15:23:49.374128 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.374144 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:49.374320 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:49.374665 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:49.374670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:49.374694 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:49.375160 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.375179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.376321 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.376327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.378324 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:49.378340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.379768 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.379774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:49.379946 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.379969 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:23:49.380068 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:23:49.380197 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":92,"thread_start_us":535,"threads_started":1}
I20260925 15:23:49.381471 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b447a9e3ff78451eac2ed6b7392c0012" candidate_uuid: "d94df4f15b644171b6074292bf80afc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" is_pre_election: true
I20260925 15:23:49.381682 29122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b447a9e3ff78451eac2ed6b7392c0012" candidate_uuid: "d94df4f15b644171b6074292bf80afc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdab9ca973374703b4e08692e05af645" is_pre_election: true
I20260925 15:23:49.382009 29383 raft_consensus.cc:2454] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d94df4f15b644171b6074292bf80afc5 in term 0.
I20260925 15:23:49.382228 29122 raft_consensus.cc:2454] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d94df4f15b644171b6074292bf80afc5 in term 0.
I20260925 15:23:49.382745 29186 leader_election.cc:302] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f63d2aef76c4d5494a15dfb09d9cd13, d94df4f15b644171b6074292bf80afc5; no voters:
I20260925 15:23:49.383111 29454 raft_consensus.cc:2781] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:49.383257 29454 raft_consensus.cc:482] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:49.383356 29454 raft_consensus.cc:3037] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:49.385775 29454 raft_consensus.cc:504] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.386314 29454 leader_election.cc:288] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [CANDIDATE]: Term 1 election: Requested vote from peers bdab9ca973374703b4e08692e05af645 (127.8.226.65:46527), 9f63d2aef76c4d5494a15dfb09d9cd13 (127.8.226.67:34075)
I20260925 15:23:49.386646 29122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b447a9e3ff78451eac2ed6b7392c0012" candidate_uuid: "d94df4f15b644171b6074292bf80afc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdab9ca973374703b4e08692e05af645"
I20260925 15:23:49.386694 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b447a9e3ff78451eac2ed6b7392c0012" candidate_uuid: "d94df4f15b644171b6074292bf80afc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13"
I20260925 15:23:49.386828 29122 raft_consensus.cc:3037] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:49.386936 29383 raft_consensus.cc:3037] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:49.389114 29122 raft_consensus.cc:2454] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d94df4f15b644171b6074292bf80afc5 in term 1.
I20260925 15:23:49.389140 29383 raft_consensus.cc:2454] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d94df4f15b644171b6074292bf80afc5 in term 1.
I20260925 15:23:49.389504 29188 leader_election.cc:302] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdab9ca973374703b4e08692e05af645, d94df4f15b644171b6074292bf80afc5; no voters:
I20260925 15:23:49.389832 29454 raft_consensus.cc:2781] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:49.390269 29454 raft_consensus.cc:686] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 1 LEADER]: Becoming Leader. State: Replica: d94df4f15b644171b6074292bf80afc5, State: Running, Role: LEADER
I20260925 15:23:49.390713 29454 consensus_queue.cc:237] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } }
I20260925 15:23:49.395368 28981 catalog_manager.cc:5495] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d94df4f15b644171b6074292bf80afc5 (127.8.226.66). New cstate: current_term: 1 leader_uuid: "d94df4f15b644171b6074292bf80afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d94df4f15b644171b6074292bf80afc5" member_type: VOTER last_known_addr { host: "127.8.226.66" port: 45873 } health_report { overall_health: HEALTHY } } }
I20260925 15:23:49.418792 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.414179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.8.226.66:45873)
0925 15:23:49.414343 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.414347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.414364 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:23:49.414488 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.414491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.414577 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.414703 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:49.414996 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.415663 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.416936 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.417391 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.417478 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.417896 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:49.418382 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:49.418482 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.418542 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.418597 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:23:49.426440 29383 raft_consensus.cc:1261] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 1 FOLLOWER]: Refusing update from remote peer d94df4f15b644171b6074292bf80afc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:49.426441 29122 raft_consensus.cc:1261] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 1 FOLLOWER]: Refusing update from remote peer d94df4f15b644171b6074292bf80afc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:49.427331 29457 consensus_queue.cc:1035] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f63d2aef76c4d5494a15dfb09d9cd13" member_type: VOTER last_known_addr { host: "127.8.226.67" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:49.427621 29454 consensus_queue.cc:1035] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdab9ca973374703b4e08692e05af645" member_type: VOTER last_known_addr { host: "127.8.226.65" port: 46527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:49.441740 29461 mvcc.cc:205] Tried to move back new op lower bound from 7333272901321785344 to 7333272901188321280. Current Snapshot: MvccSnapshot[applied={T|T < 7333272901321785344}]
I20260925 15:23:49.442768 29462 mvcc.cc:205] Tried to move back new op lower bound from 7333272901321785344 to 7333272901188321280. Current Snapshot: MvccSnapshot[applied={T|T < 7333272901321785344 or (T in {7333272901321785344})}]
I20260925 15:23:49.453146 28980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:49.453465 28980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:49.457126 28980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fc6f9582c6b46e68159185dc0a9dca3: Sending DeleteTablet for 3 replicas of tablet b447a9e3ff78451eac2ed6b7392c0012
I20260925 15:23:49.458102 29102 tablet_service.cc:1514] Processing DeleteTablet for tablet b447a9e3ff78451eac2ed6b7392c0012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:49 UTC) from {username='slave'} at 127.0.0.1:41446
I20260925 15:23:49.458204 29363 tablet_service.cc:1514] Processing DeleteTablet for tablet b447a9e3ff78451eac2ed6b7392c0012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:49 UTC) from {username='slave'} at 127.0.0.1:42118
I20260925 15:23:49.459326 29233 tablet_service.cc:1514] Processing DeleteTablet for tablet b447a9e3ff78451eac2ed6b7392c0012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:49 UTC) from {username='slave'} at 127.0.0.1:37650
I20260925 15:23:49.461175 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29040
I20260925 15:23:49.472671 29471 tablet_replica.cc:330] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: stopping tablet replica
I20260925 15:23:49.473351 29471 raft_consensus.cc:2229] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:49.473816 29469 tablet_replica.cc:330] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: stopping tablet replica
I20260925 15:23:49.474332 29471 raft_consensus.cc:2258] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:49.475063 29469 raft_consensus.cc:2229] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:49.477070 29471 ts_tablet_manager.cc:1905] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:49.477552 29469 raft_consensus.cc:2258] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:49.481963 29470 tablet_replica.cc:330] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: stopping tablet replica
I20260925 15:23:49.482378 29470 raft_consensus.cc:2229] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:49.482875 29470 raft_consensus.cc:2258] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:49.484817 29469 ts_tablet_manager.cc:1905] T b447a9e3ff78451eac2ed6b7392c0012 P bdab9ca973374703b4e08692e05af645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:49.485642 29471 ts_tablet_manager.cc:1918] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:49.485724 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.462215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.8.226.65:46527)
0925 15:23:49.462380 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.462387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.475201 (+ 12814us) server_negotiation.cc:407] Connection header received
0925 15:23:49.475482 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.475487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.475610 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.475732 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:49.476483 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.477640 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.480063 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.480643 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.480763 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.483512 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:49.483524 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:49.483541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:49.484315 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:49.484372 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:49.484400 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:49.484511 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.485229 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.485366 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:23:49.486037 29471 log.cc:1196] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-1/wal/wals/b447a9e3ff78451eac2ed6b7392c0012
I20260925 15:23:49.486654 29471 ts_tablet_manager.cc:1939] T b447a9e3ff78451eac2ed6b7392c0012 P d94df4f15b644171b6074292bf80afc5: Deleting consensus metadata
I20260925 15:23:49.486821 29162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:49.487222 29470 ts_tablet_manager.cc:1905] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:49.488441 28966 catalog_manager.cc:4841] TS d94df4f15b644171b6074292bf80afc5 (127.8.226.66:45873): tablet b447a9e3ff78451eac2ed6b7392c0012 (table test-table [id=f8aed80b5ca8490688e8f9bb95a08b2f]) successfully deleted
I20260925 15:23:49.494880 29470 ts_tablet_manager.cc:1918] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:49.495035 29470 log.cc:1196] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Deleting WAL directory at /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349523192742-9097-0/minicluster-data/ts-2/wal/wals/b447a9e3ff78451eac2ed6b7392c0012
I20260925 15:23:49.495558 29470 ts_tablet_manager.cc:1939] T b447a9e3ff78451eac2ed6b7392c0012 P 9f63d2aef76c4d5494a15dfb09d9cd13: Deleting consensus metadata
I20260925 15:23:49.497445 28965 catalog_manager.cc:4841] TS 9f63d2aef76c4d5494a15dfb09d9cd13 (127.8.226.67:34075): tablet b447a9e3ff78451eac2ed6b7392c0012 (table test-table [id=f8aed80b5ca8490688e8f9bb95a08b2f]) successfully deleted
I20260925 15:23:49.580058 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29040
W20260925 15:23:49.597379 28967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.226.65:46527 (error 108)
I20260925 15:23:49.597532 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29171
I20260925 15:23:49.599435 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.598971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41470)
0925 15:23:49.599157 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.599283 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 15:23:49.599742 28967 catalog_manager.cc:4584] TS bdab9ca973374703b4e08692e05af645 (127.8.226.65:46527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b447a9e3ff78451eac2ed6b7392c0012: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
I20260925 15:23:49.606082 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.598642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.8.226.66:45873)
0925 15:23:49.598820 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.598824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.598844 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:23:49.598897 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.598900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.598972 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.599105 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:49.599646 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.600853 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.602154 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.602864 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.602991 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.604413 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:49.604427 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:49.604445 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:49.605322 (+ 877us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:49.605384 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:49.605408 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:49.605517 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.605603 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.605703 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:23:49.606863 29293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:49.653548 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.653094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41476)
0925 15:23:49.653298 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.653387 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 15:23:49.654520 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.654217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41480)
0925 15:23:49.654340 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.654416 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:23:49.715505 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.714705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41496)
0925 15:23:49.715124 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.715258 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 15:23:49.716580 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.716182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41504)
0925 15:23:49.716328 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.716439 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:23:49.718448 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29171
I20260925 15:23:49.736907 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29301
I20260925 15:23:49.749663 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.737736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.8.226.67:34075)
0925 15:23:49.737902 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:49.737907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:49.737948 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:23:49.738149 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:49.738152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:49.738240 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:49.738321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:49.738726 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.739443 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.741233 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:49.741709 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:49.741807 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:49.748166 (+ 6359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:49.748175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:49.748190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:49.748810 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:49.748862 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:49.748897 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:49.749069 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:49.749217 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:23:49.749341 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:23:49.750721 29423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:49.797123 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.796610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41512)
0925 15:23:49.796834 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.796918 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 15:23:49.798123 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.797750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41522)
0925 15:23:49.797957 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.798023 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 15:23:49.859148 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29301
I20260925 15:23:49.875588 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 28950
I20260925 15:23:49.876593 29011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:49.984886 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:49.984226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.65:46527 (local address 127.0.0.1:41532)
0925 15:23:49.984470 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:49.984582 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.65:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:23:49.985059 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 28950
2026-09-25T15:23:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2724 ms)
[----------] 6 tests from EncryptionPolicyTest (21151 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:50 dist-test-slave-8vw7 krb5kdc[29480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:50 dist-test-slave-8vw7 krb5kdc[29480](info): set up 2 sockets
Sep 25 15:23:50 dist-test-slave-8vw7 krb5kdc[29480](info): commencing operation
krb5kdc: starting...
W20260925 15:23:51.085541 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:51 dist-test-slave-8vw7 krb5kdc[29480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:23:51.315584 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:51.634141 29493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:51.634470 29493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:51.634647 29493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:51.641594 29493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:51.641737 29493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:51.641840 29493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:51.641947 29493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:51.642031 29493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:51.652613 29493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:51.653998 29493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:51.655400 29493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:51.667200 29498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:51.667162 29501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:51.667239 29499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:51.667893 29493 server_base.cc:1034] running on GCE node
I20260925 15:23:51.668651 29493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:51.670149 29493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:51.671386 29493 hybrid_clock.cc:648] HybridClock initialized: now 1790349831671343 us; error 58 us; skew 500 ppm
I20260925 15:23:51.671741 29493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:51 dist-test-slave-8vw7 krb5kdc[29480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:51.674928 29493 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:23:51.676390 29493 webserver.cc:460] Webserver started at http://127.0.0.1:36419/ using document root <none> and password file <none>
I20260925 15:23:51.677342 29493 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:51.677492 29493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:51.677860 29493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:51.682307 29493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "66337a6623cb483ba34659d3f03a1140"
format_stamp: "Formatted at 2026-09-25 15:23:51 on dist-test-slave-8vw7"
server_key: "e5af15f1332b536c8e99d1cb73e11cf1"
server_key_iv: "9e51d062901b59eac741d00d00056fcb"
server_key_version: "encryptionkey@0"
I20260925 15:23:51.683373 29493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "66337a6623cb483ba34659d3f03a1140"
format_stamp: "Formatted at 2026-09-25 15:23:51 on dist-test-slave-8vw7"
server_key: "e5af15f1332b536c8e99d1cb73e11cf1"
server_key_iv: "9e51d062901b59eac741d00d00056fcb"
server_key_version: "encryptionkey@0"
I20260925 15:23:51.688462 29493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:23:51.691766 29508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:51.693130 29493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:51.693298 29493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "66337a6623cb483ba34659d3f03a1140"
format_stamp: "Formatted at 2026-09-25 15:23:51 on dist-test-slave-8vw7"
server_key: "e5af15f1332b536c8e99d1cb73e11cf1"
server_key_iv: "9e51d062901b59eac741d00d00056fcb"
server_key_version: "encryptionkey@0"
I20260925 15:23:51.693476 29493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:51.707182 29493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:51.711272 29493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:51.711512 29493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:51.728932 29493 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36307
I20260925 15:23:51.728987 29559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36307 every 8 connection(s)
I20260925 15:23:51.730360 29493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:51.733819 29560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:51.737532 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29493
I20260925 15:23:51.737751 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:51.738590 9097 external_mini_cluster.cc:1428] Setting key cf853fdb19017946a4b3fbe159cb36db
I20260925 15:23:51.747097 29560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140: Bootstrap starting.
I20260925 15:23:51.750353 29560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:51.751328 29560 log.cc:824] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:23:51 dist-test-slave-8vw7 krb5kdc[29480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:23:51.754222 29560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140: No bootstrap required, opened a new log
I20260925 15:23:51.761956 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:51.740392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36963 (local address 127.0.0.1:36307)
0925 15:23:51.741084 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:51.741098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:51.741144 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:23:51.743003 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:51.743029 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:51.743523 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:51.743952 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:51.745286 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:51.747173 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:51.748445 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:51.749437 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:51.749632 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:51.753684 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:51.753694 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:51.753720 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:51.753761 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:51.757568 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:51.758293 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:51.758300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:51.758307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:51.758507 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:51.758964 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:51.758970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:51.758973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:51.759629 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:51.759966 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:51.760300 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:23:51.760623 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":288,"threads_started":1}
I20260925 15:23:51.764564 29560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66337a6623cb483ba34659d3f03a1140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260925 15:23:51.764896 29560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:51.765060 29560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66337a6623cb483ba34659d3f03a1140, State: Initialized, Role: FOLLOWER
I20260925 15:23:51.765709 29560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66337a6623cb483ba34659d3f03a1140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260925 15:23:51.765956 29560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:51.765973 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:23:51.766095 29560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:51.766279 29560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:51.768328 29560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66337a6623cb483ba34659d3f03a1140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260925 15:23:51.768787 29560 leader_election.cc:302] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66337a6623cb483ba34659d3f03a1140; no voters:
I20260925 15:23:51.769460 29560 leader_election.cc:288] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:51.769584 29568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:51.771301 29568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [term 1 LEADER]: Becoming Leader. State: Replica: 66337a6623cb483ba34659d3f03a1140, State: Running, Role: LEADER
I20260925 15:23:51.771696 29560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:51.771917 29568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66337a6623cb483ba34659d3f03a1140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260925 15:23:51.775123 29570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66337a6623cb483ba34659d3f03a1140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66337a6623cb483ba34659d3f03a1140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } } }
I20260925 15:23:51.775593 29570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:51.775552 29571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66337a6623cb483ba34659d3f03a1140. Latest consensus state: current_term: 1 leader_uuid: "66337a6623cb483ba34659d3f03a1140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66337a6623cb483ba34659d3f03a1140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } } }
I20260925 15:23:51.775913 29571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:51.783090 29576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:51.788100 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:51.771219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54879 (local address 127.0.0.1:36307)
0925 15:23:51.771515 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:51.771520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:51.771538 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:23:51.771583 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:51.771587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:51.771651 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:51.771775 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:51.773763 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:51.775284 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:51.776784 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:51.777496 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:51.777609 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:51.781145 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:51.781158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:51.781178 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:51.781228 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:51.784409 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:51.785422 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:51.785429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:51.785435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:51.785606 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:51.786245 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:51.786250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:51.786254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:51.786656 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:51.786956 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:51.787287 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:23:51.787462 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:23:51.789738 29576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:51.798748 29576 catalog_manager.cc:1347] Generated new cluster ID: 41d3e45ab6e445c0bd01edd806682437
I20260925 15:23:51.798849 29576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:51.839494 29576 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:51.841312 29576 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:51.852809 29576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66337a6623cb483ba34659d3f03a1140: Generated new TSK 0
I20260925 15:23:51.853868 29576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:51.867552 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29493
I20260925 15:23:51.868676 29555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:51.959555 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29493
W20260925 15:23:51.977241 29478 connection.cc:535] client connection to 127.0.0.1:36307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36307): Connection reset by peer (error 104)
2026-09-25T15:23:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:52 dist-test-slave-8vw7 krb5kdc[29592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:52 dist-test-slave-8vw7 krb5kdc[29592](info): set up 2 sockets
Sep 25 15:23:52 dist-test-slave-8vw7 krb5kdc[29592](info): commencing operation
krb5kdc: starting...
W20260925 15:23:53.049351 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:53 dist-test-slave-8vw7 krb5kdc[29592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:23:53.284861 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:53.601718 29605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:53.601997 29605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:53.602159 29605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:53.609077 29605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:53.609197 29605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:53.609297 29605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:53.609400 29605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:53.609477 29605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:53.618743 29605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:53.619854 29605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:53.621121 29605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:53.632535 29613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:53.632606 29610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:53.632575 29611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:53.633354 29605 server_base.cc:1034] running on GCE node
I20260925 15:23:53.634138 29605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:53.635651 29605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:53.636853 29605 hybrid_clock.cc:648] HybridClock initialized: now 1790349833636800 us; error 46 us; skew 500 ppm
I20260925 15:23:53.637193 29605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:53 dist-test-slave-8vw7 krb5kdc[29592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:53.640158 29605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:23:53.641479 29605 webserver.cc:460] Webserver started at http://127.0.0.1:44455/ using document root <none> and password file <none>
I20260925 15:23:53.642066 29605 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:53.642158 29605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:53.642383 29605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:53.645177 29605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "26efab095aca4269ac748456c41fef43"
format_stamp: "Formatted at 2026-09-25 15:23:53 on dist-test-slave-8vw7"
server_key: "df2661746e07ada6d2a0898a998753e6"
server_key_iv: "78220c6ca0dc5a10b9ddbebda32a0063"
server_key_version: "encryptionkey@0"
I20260925 15:23:53.645920 29605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "26efab095aca4269ac748456c41fef43"
format_stamp: "Formatted at 2026-09-25 15:23:53 on dist-test-slave-8vw7"
server_key: "df2661746e07ada6d2a0898a998753e6"
server_key_iv: "78220c6ca0dc5a10b9ddbebda32a0063"
server_key_version: "encryptionkey@0"
I20260925 15:23:53.650624 29605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:23:53.653961 29620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:53.655241 29605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:53.655370 29605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "26efab095aca4269ac748456c41fef43"
format_stamp: "Formatted at 2026-09-25 15:23:53 on dist-test-slave-8vw7"
server_key: "df2661746e07ada6d2a0898a998753e6"
server_key_iv: "78220c6ca0dc5a10b9ddbebda32a0063"
server_key_version: "encryptionkey@0"
I20260925 15:23:53.655520 29605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:53.678571 29605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:53.682976 29605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:53.683197 29605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:53.700981 29605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39559
I20260925 15:23:53.700999 29671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39559 every 8 connection(s)
I20260925 15:23:53.702311 29605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:53.705746 29672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:53.706699 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29605
I20260925 15:23:53.706876 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:53.707325 9097 external_mini_cluster.cc:1428] Setting key f50c4b5e442d878cf88aa3a0b3ad79cc
I20260925 15:23:53.719602 29672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43: Bootstrap starting.
I20260925 15:23:53.721487 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:53.709126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44983 (local address 127.0.0.1:39559)
0925 15:23:53.709878 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:53.709892 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:53.709941 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:23:53.712139 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:53.712163 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:53.712631 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:53.713066 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:53.713549 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:53.715370 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:53.716608 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:53.717723 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:53.717880 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:53.718923 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:53.718933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:53.718948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:53.719603 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:53.719707 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:53.719737 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:53.719888 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:53.720103 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:23:53.720249 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":310,"threads_started":1}
I20260925 15:23:53.723083 29672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:53.724260 29672 log.cc:824] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:53.725059 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:23:53.727176 29672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43: No bootstrap required, opened a new log
I20260925 15:23:53.736505 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:53.730133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53009 (local address 127.0.0.1:39559)
0925 15:23:53.730317 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:53.730323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:53.730790 (+ 467us) server_negotiation.cc:407] Connection header received
0925 15:23:53.730928 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:53.730938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:53.731012 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:53.731118 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:53.731540 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:53.732663 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:53.733961 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:53.734680 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:53.734818 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:53.735787 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:53.735793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:53.735797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:53.735816 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:53.735873 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:53.735891 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:53.735997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:53.736152 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:23:53.736206 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:23:53.736753 29672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26efab095aca4269ac748456c41fef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39559 } }
I20260925 15:23:53.737246 29672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:53.737368 29672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26efab095aca4269ac748456c41fef43, State: Initialized, Role: FOLLOWER
I20260925 15:23:53.737959 29672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26efab095aca4269ac748456c41fef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39559 } }
I20260925 15:23:53.738152 29672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:53.738247 29672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:53.738369 29672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:53.740888 29672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26efab095aca4269ac748456c41fef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39559 } }
I20260925 15:23:53.741374 29672 leader_election.cc:302] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26efab095aca4269ac748456c41fef43; no voters:
I20260925 15:23:53.741854 29672 leader_election.cc:288] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:53.741945 29682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:53.743750 29672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:53.743856 29682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [term 1 LEADER]: Becoming Leader. State: Replica: 26efab095aca4269ac748456c41fef43, State: Running, Role: LEADER
I20260925 15:23:53.744319 29682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26efab095aca4269ac748456c41fef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39559 } }
I20260925 15:23:53.747349 29684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26efab095aca4269ac748456c41fef43. Latest consensus state: current_term: 1 leader_uuid: "26efab095aca4269ac748456c41fef43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26efab095aca4269ac748456c41fef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39559 } } }
I20260925 15:23:53.747761 29684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:53.748085 29683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26efab095aca4269ac748456c41fef43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26efab095aca4269ac748456c41fef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39559 } } }
I20260925 15:23:53.748385 29683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43 [sys.catalog]: This master's current role is: LEADER
W20260925 15:23:53.755786 29698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:23:53.755882 29698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:23:53.756043 29694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:53.757803 29694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:53.765434 29694 catalog_manager.cc:1347] Generated new cluster ID: 91473e461f7e468f8ab3f20b407ac580
I20260925 15:23:53.765585 29694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:53.777812 29694 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:53.778954 29694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:53.792273 29694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26efab095aca4269ac748456c41fef43: Generated new TSK 0
I20260925 15:23:53.793254 29694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:53.806051 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29605
W20260925 15:23:53.807231 29667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44983
W20260925 15:23:53.807937 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:23:53.808107 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29605
W20260925 15:23:53.824437 29590 connection.cc:535] client connection to 127.0.0.1:39559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39559): Connection reset by peer (error 104)
2026-09-25T15:23:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:53 dist-test-slave-8vw7 krb5kdc[29703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:53 dist-test-slave-8vw7 krb5kdc[29703](info): set up 2 sockets
Sep 25 15:23:53 dist-test-slave-8vw7 krb5kdc[29703](info): commencing operation
krb5kdc: starting...
W20260925 15:23:54.905400 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:55 dist-test-slave-8vw7 krb5kdc[29703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:23:55.138159 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:55.460170 29716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:55.460486 29716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:55.460644 29716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:55.467684 29716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:55.467818 29716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:55.467964 29716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:55.468060 29716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:55.468144 29716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:55.478154 29716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:55.479403 29716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:55.480654 29716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:55.493379 29722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:55.493495 29721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:55.493588 29724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:55.494112 29716 server_base.cc:1034] running on GCE node
I20260925 15:23:55.494930 29716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:55.496290 29716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:55.497500 29716 hybrid_clock.cc:648] HybridClock initialized: now 1790349835497462 us; error 29 us; skew 500 ppm
I20260925 15:23:55.497859 29716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:55 dist-test-slave-8vw7 krb5kdc[29703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:55.506853 29716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:23:55.508358 29716 webserver.cc:460] Webserver started at http://127.0.0.1:33173/ using document root <none> and password file <none>
I20260925 15:23:55.509097 29716 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:55.509229 29716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:55.509532 29716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:55.512642 29716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "1525f5120d6441ef8c9059673e80c842"
format_stamp: "Formatted at 2026-09-25 15:23:55 on dist-test-slave-8vw7"
server_key: "82e290d1f42207a8646aad0a00ff312c"
server_key_iv: "c049107b08e2a6d688e9ea353c84536a"
server_key_version: "encryptionkey@0"
I20260925 15:23:55.513453 29716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "1525f5120d6441ef8c9059673e80c842"
format_stamp: "Formatted at 2026-09-25 15:23:55 on dist-test-slave-8vw7"
server_key: "82e290d1f42207a8646aad0a00ff312c"
server_key_iv: "c049107b08e2a6d688e9ea353c84536a"
server_key_version: "encryptionkey@0"
I20260925 15:23:55.518532 29716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:23:55.522197 29731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:55.523480 29716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:23:55.523658 29716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "1525f5120d6441ef8c9059673e80c842"
format_stamp: "Formatted at 2026-09-25 15:23:55 on dist-test-slave-8vw7"
server_key: "82e290d1f42207a8646aad0a00ff312c"
server_key_iv: "c049107b08e2a6d688e9ea353c84536a"
server_key_version: "encryptionkey@0"
I20260925 15:23:55.523870 29716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:55.529376 29716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:55.533317 29716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:55.533594 29716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:55.552235 29716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44555
I20260925 15:23:55.552281 29782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44555 every 8 connection(s)
I20260925 15:23:55.553596 29716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:55.557099 29783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:55.561112 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29716
I20260925 15:23:55.561283 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:55.561713 9097 external_mini_cluster.cc:1428] Setting key a8c8bafbde082d824e4087202ad51b06
I20260925 15:23:55.568658 29783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842: Bootstrap starting.
I20260925 15:23:55.570453 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:55.563372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43523 (local address 127.0.0.1:44555)
0925 15:23:55.564150 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:55.564164 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:55.564223 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:23:55.566903 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:55.566928 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:55.567315 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:55.567707 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:55.568616 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:55.568623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:55.568640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:55.568664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:55.568736 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:55.568771 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:55.568997 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:55.569118 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:23:55.569300 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":300,"threads_started":1}
I20260925 15:23:55.571980 29783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:55.573167 29783 log.cc:824] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:55.573751 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:23:55.575588 29783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842: No bootstrap required, opened a new log
I20260925 15:23:55.580176 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:55.577948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43915 (local address 127.0.0.1:44555)
0925 15:23:55.578111 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:55.578117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:55.578411 (+ 294us) server_negotiation.cc:407] Connection header received
0925 15:23:55.578532 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:55.578537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:55.578591 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:55.578690 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:55.579542 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:55.579547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:55.579554 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:55.579575 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:55.579623 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:55.579646 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:55.579758 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:55.579826 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:23:55.579884 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:23:55.583407 29783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1525f5120d6441ef8c9059673e80c842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260925 15:23:55.583747 29783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:55.583827 29783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1525f5120d6441ef8c9059673e80c842, State: Initialized, Role: FOLLOWER
I20260925 15:23:55.584275 29783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1525f5120d6441ef8c9059673e80c842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260925 15:23:55.584471 29783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:55.584561 29783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:55.584679 29783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:55.586957 29783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1525f5120d6441ef8c9059673e80c842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260925 15:23:55.587354 29783 leader_election.cc:302] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1525f5120d6441ef8c9059673e80c842; no voters:
I20260925 15:23:55.587853 29783 leader_election.cc:288] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:55.587931 29793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:55.588598 29793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [term 1 LEADER]: Becoming Leader. State: Replica: 1525f5120d6441ef8c9059673e80c842, State: Running, Role: LEADER
I20260925 15:23:55.589143 29793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1525f5120d6441ef8c9059673e80c842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260925 15:23:55.589993 29783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:55.592273 29794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1525f5120d6441ef8c9059673e80c842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1525f5120d6441ef8c9059673e80c842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } } }
I20260925 15:23:55.592299 29795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1525f5120d6441ef8c9059673e80c842. Latest consensus state: current_term: 1 leader_uuid: "1525f5120d6441ef8c9059673e80c842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1525f5120d6441ef8c9059673e80c842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } } }
I20260925 15:23:55.592800 29794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:55.592797 29795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:55.604380 29801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:55.609541 29801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:55.617589 29801 catalog_manager.cc:1347] Generated new cluster ID: cf6bd93c601f45eab9dfbf52ee91a190
I20260925 15:23:55.617709 29801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:55.630264 29801 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:55.630733 29801 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:55.648422 29801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1525f5120d6441ef8c9059673e80c842: Generated new TSK 0
I20260925 15:23:55.649421 29801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:55.656744 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29716
W20260925 15:23:55.657866 29778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43523
W20260925 15:23:55.658499 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29716: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:23:55.658641 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29716
2026-09-25T15:23:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1850 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:55 dist-test-slave-8vw7 krb5kdc[29814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:55 dist-test-slave-8vw7 krb5kdc[29814](info): set up 2 sockets
Sep 25 15:23:55 dist-test-slave-8vw7 krb5kdc[29814](info): commencing operation
krb5kdc: starting...
W20260925 15:23:56.733408 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:56 dist-test-slave-8vw7 krb5kdc[29814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:23:56.957132 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:57.274736 29827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:57.275034 29827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:57.275200 29827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:57.282131 29827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:57.282258 29827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:57.282341 29827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:57.282464 29827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:57.282526 29827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:57.291887 29827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:57.293112 29827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:57.294376 29827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:57.305554 29832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:57.305480 29835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:57.305524 29833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:57.306591 29827 server_base.cc:1034] running on GCE node
I20260925 15:23:57.307376 29827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:57.308650 29827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:57.309880 29827 hybrid_clock.cc:648] HybridClock initialized: now 1790349837309827 us; error 32 us; skew 500 ppm
I20260925 15:23:57.310201 29827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:57 dist-test-slave-8vw7 krb5kdc[29814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:57.313252 29827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:23:57.314482 29827 webserver.cc:460] Webserver started at http://127.0.0.1:45299/ using document root <none> and password file <none>
I20260925 15:23:57.315070 29827 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:57.315156 29827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:57.315374 29827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:57.318125 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "3bff1ba65f434e839b298a81e9cd264c"
format_stamp: "Formatted at 2026-09-25 15:23:57 on dist-test-slave-8vw7"
server_key: "87c36af8b798b551518acc75d6407407"
server_key_iv: "80a3a852a262e2772c1b0bb97a5fce39"
server_key_version: "encryptionkey@0"
I20260925 15:23:57.318924 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "3bff1ba65f434e839b298a81e9cd264c"
format_stamp: "Formatted at 2026-09-25 15:23:57 on dist-test-slave-8vw7"
server_key: "87c36af8b798b551518acc75d6407407"
server_key_iv: "80a3a852a262e2772c1b0bb97a5fce39"
server_key_version: "encryptionkey@0"
I20260925 15:23:57.323745 29827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:23:57.326987 29842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:57.328420 29827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:57.328554 29827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "3bff1ba65f434e839b298a81e9cd264c"
format_stamp: "Formatted at 2026-09-25 15:23:57 on dist-test-slave-8vw7"
server_key: "87c36af8b798b551518acc75d6407407"
server_key_iv: "80a3a852a262e2772c1b0bb97a5fce39"
server_key_version: "encryptionkey@0"
I20260925 15:23:57.328747 29827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:57.341523 29827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:57.345623 29827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:57.345834 29827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:57.363409 29827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36751
I20260925 15:23:57.363440 29893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36751 every 8 connection(s)
I20260925 15:23:57.364743 29827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:57.366055 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29827
I20260925 15:23:57.366235 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:57.366709 9097 external_mini_cluster.cc:1428] Setting key ade940d29db29f7b7ba0e65ffc6a5e2d
I20260925 15:23:57.368901 29894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:57.379436 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c: Bootstrap starting.
Sep 25 15:23:57 dist-test-slave-8vw7 krb5kdc[29814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:23:57.382651 29894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:57.383617 29894 log.cc:824] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:57.386237 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c: No bootstrap required, opened a new log
I20260925 15:23:57.387530 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:57.368392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54931 (local address 127.0.0.1:36751)
0925 15:23:57.369212 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:57.369223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:57.369260 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:23:57.370828 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:57.370847 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:57.371226 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:57.371616 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:57.373348 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:57.374686 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:57.375894 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:57.376556 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:57.376765 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:57.380760 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:57.380767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:57.380783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:57.380820 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:57.383861 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:57.384481 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:57.384496 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:57.384501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:57.384674 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:57.385083 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:57.385088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:57.385091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:57.385532 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:57.385821 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:57.386267 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 15:23:57.386665 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":330,"threads_started":1}
I20260925 15:23:57.391284 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:23:57.394598 29894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ba65f434e839b298a81e9cd264c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260925 15:23:57.394958 29894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:57.395056 29894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bff1ba65f434e839b298a81e9cd264c, State: Initialized, Role: FOLLOWER
I20260925 15:23:57.395666 29894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ba65f434e839b298a81e9cd264c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260925 15:23:57.395941 29894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:57.396086 29894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:57.396250 29894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:57.398705 29894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ba65f434e839b298a81e9cd264c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260925 15:23:57.399255 29894 leader_election.cc:302] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bff1ba65f434e839b298a81e9cd264c; no voters:
I20260925 15:23:57.399837 29894 leader_election.cc:288] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:57.399967 29904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:57.400581 29904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [term 1 LEADER]: Becoming Leader. State: Replica: 3bff1ba65f434e839b298a81e9cd264c, State: Running, Role: LEADER
I20260925 15:23:57.401262 29904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ba65f434e839b298a81e9cd264c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260925 15:23:57.404215 29894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:57.403975 29906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bff1ba65f434e839b298a81e9cd264c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ba65f434e839b298a81e9cd264c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } } }
I20260925 15:23:57.404482 29906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:57.405401 29908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:57.405274 29905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bff1ba65f434e839b298a81e9cd264c. Latest consensus state: current_term: 1 leader_uuid: "3bff1ba65f434e839b298a81e9cd264c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ba65f434e839b298a81e9cd264c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } } }
I20260925 15:23:57.405632 29905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:57.414495 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:57.395604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49521 (local address 127.0.0.1:36751)
0925 15:23:57.395773 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:57.395779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:57.396259 (+ 480us) server_negotiation.cc:407] Connection header received
0925 15:23:57.396430 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:57.396436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:57.396517 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:57.396621 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:57.397998 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:57.399077 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:57.400904 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:57.401627 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:57.401725 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:57.405104 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:57.405112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:57.405122 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:57.405176 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:57.409739 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:57.410847 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:57.410855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:57.410861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:57.411119 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:57.411662 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:57.411666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:57.411670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:57.412122 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:57.412468 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:57.413790 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0925 15:23:57.414064 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:23:57.418520 29908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:57.431005 29908 catalog_manager.cc:1347] Generated new cluster ID: 6ce5cae86d7942b7b563149efd174cfb
I20260925 15:23:57.431099 29908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:57.439847 29908 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:57.441085 29908 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:57.450423 29908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bff1ba65f434e839b298a81e9cd264c: Generated new TSK 0
I20260925 15:23:57.451112 29908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:57.456748 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29827
I20260925 15:23:57.457927 29889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:23:57.545007 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29827
2026-09-25T15:23:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:57 dist-test-slave-8vw7 krb5kdc[29926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:57 dist-test-slave-8vw7 krb5kdc[29926](info): set up 2 sockets
Sep 25 15:23:57 dist-test-slave-8vw7 krb5kdc[29926](info): commencing operation
krb5kdc: starting...
W20260925 15:23:58.617352 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:23:58 dist-test-slave-8vw7 krb5kdc[29926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:23:58.871955 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:59.191433 29939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:59.191708 29939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:59.191798 29939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:59.198763 29939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:59.198866 29939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:59.198912 29939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:59.198972 29939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:59.199009 29939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:59.208217 29939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:23:59.209357 29939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:59.210541 29939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:59.221426 29947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:59.221489 29945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:59.221683 29944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:59.222697 29939 server_base.cc:1034] running on GCE node
I20260925 15:23:59.223414 29939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:59.224745 29939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:59.225960 29939 hybrid_clock.cc:648] HybridClock initialized: now 1790349839225894 us; error 61 us; skew 500 ppm
I20260925 15:23:59.226301 29939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:23:59 dist-test-slave-8vw7 krb5kdc[29926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:59.229409 29939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:23:59.230809 29939 webserver.cc:460] Webserver started at http://127.0.0.1:38555/ using document root <none> and password file <none>
I20260925 15:23:59.231472 29939 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:59.231616 29939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:59.231909 29939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:59.234836 29939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "ad633de770f344a7bc13648287971329"
format_stamp: "Formatted at 2026-09-25 15:23:59 on dist-test-slave-8vw7"
server_key: "c135cfc4e10894d5bf43fee20bdc2476"
server_key_iv: "4762299893844f9dc5ebab56a0e4145c"
server_key_version: "encryptionkey@0"
I20260925 15:23:59.235569 29939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "ad633de770f344a7bc13648287971329"
format_stamp: "Formatted at 2026-09-25 15:23:59 on dist-test-slave-8vw7"
server_key: "c135cfc4e10894d5bf43fee20bdc2476"
server_key_iv: "4762299893844f9dc5ebab56a0e4145c"
server_key_version: "encryptionkey@0"
I20260925 15:23:59.240347 29939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:23:59.243516 29954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:59.244787 29939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:23:59.245004 29939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "ad633de770f344a7bc13648287971329"
format_stamp: "Formatted at 2026-09-25 15:23:59 on dist-test-slave-8vw7"
server_key: "c135cfc4e10894d5bf43fee20bdc2476"
server_key_iv: "4762299893844f9dc5ebab56a0e4145c"
server_key_version: "encryptionkey@0"
I20260925 15:23:59.245216 29939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:59.260128 29939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:59.264075 29939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:59.264349 29939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:59.281836 29939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36439
I20260925 15:23:59.281881 30005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36439 every 8 connection(s)
I20260925 15:23:59.283119 29939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:59.285590 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 29939
I20260925 15:23:59.285775 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:23:59.286206 9097 external_mini_cluster.cc:1428] Setting key eb1fe5eecb22beff9569d4c821f60e5c
I20260925 15:23:59.286885 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:59.298887 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.288020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53115 (local address 127.0.0.1:36439)
0925 15:23:59.288743 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:59.288754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:59.288791 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:23:59.290427 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:59.290446 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:59.290801 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:59.291108 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:59.291715 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.293070 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.294827 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.295457 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.295578 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:59.296751 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:59.296758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:59.296769 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:59.297348 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:59.297414 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:59.297441 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:59.297527 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:59.297761 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:23:59.297997 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":342,"threads_started":1}
I20260925 15:23:59.300175 30006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329: Bootstrap starting.
I20260925 15:23:59.302117 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:23:59.303427 30006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:59.304569 30006 log.cc:824] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:59.307654 30006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329: No bootstrap required, opened a new log
I20260925 15:23:59.313663 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.307368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56965 (local address 127.0.0.1:36439)
0925 15:23:59.307517 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:59.307522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:59.307911 (+ 389us) server_negotiation.cc:407] Connection header received
0925 15:23:59.308046 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:59.308052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:59.308138 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:59.308250 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:59.308668 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.309806 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.311012 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.311697 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.311808 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:59.312809 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:59.312815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:59.312820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:23:59.312839 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:59.312887 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:23:59.312909 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:59.313041 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:59.313182 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:23:59.313290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:23:59.316522 30006 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad633de770f344a7bc13648287971329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36439 } }
I20260925 15:23:59.316839 30006 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:59.316908 30006 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad633de770f344a7bc13648287971329, State: Initialized, Role: FOLLOWER
I20260925 15:23:59.317430 30006 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad633de770f344a7bc13648287971329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36439 } }
I20260925 15:23:59.317610 30006 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:59.317696 30006 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:59.317822 30006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:59.319950 30006 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad633de770f344a7bc13648287971329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36439 } }
I20260925 15:23:59.320462 30006 leader_election.cc:302] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad633de770f344a7bc13648287971329; no voters:
I20260925 15:23:59.321020 30006 leader_election.cc:288] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:59.321142 30016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:59.321784 30016 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [term 1 LEADER]: Becoming Leader. State: Replica: ad633de770f344a7bc13648287971329, State: Running, Role: LEADER
I20260925 15:23:59.322243 30016 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad633de770f344a7bc13648287971329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36439 } }
I20260925 15:23:59.322916 30006 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:59.324729 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad633de770f344a7bc13648287971329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad633de770f344a7bc13648287971329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36439 } } }
I20260925 15:23:59.325233 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:59.325940 30025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:59.327991 30018 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad633de770f344a7bc13648287971329. Latest consensus state: current_term: 1 leader_uuid: "ad633de770f344a7bc13648287971329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad633de770f344a7bc13648287971329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36439 } } }
I20260925 15:23:59.328425 30018 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:59.330602 30025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:59.340062 30025 catalog_manager.cc:1347] Generated new cluster ID: 2a0919cd0cab4a819e9fd9a72aad15ff
I20260925 15:23:59.340163 30025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:59.351234 30025 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:59.352924 30025 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:59.363304 30025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad633de770f344a7bc13648287971329: Generated new TSK 0
I20260925 15:23:59.364248 30025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:23:59.373755 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29939
W20260925 15:23:59.375183 30001 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53115
W20260925 15:23:59.375865 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 29939: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:23:59.376013 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 29939
2026-09-25T15:23:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:59 dist-test-slave-8vw7 krb5kdc[30037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:59 dist-test-slave-8vw7 krb5kdc[30037](info): set up 2 sockets
Sep 25 15:23:59 dist-test-slave-8vw7 krb5kdc[30037](info): commencing operation
krb5kdc: starting...
W20260925 15:24:00.457419 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:00 dist-test-slave-8vw7 krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:00.682142 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:00.996665 30050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:00.996941 30050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:00.997165 30050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:01.004283 30050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:01.004415 30050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:01.004524 30050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:01.004640 30050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:01.004737 30050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:01.014585 30050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:01.015821 30050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:01.017102 30050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:01.028656 30055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:01.028746 30058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:01.028718 30056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:01.029762 30050 server_base.cc:1034] running on GCE node
I20260925 15:24:01.030493 30050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:01.031895 30050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:01.033052 30050 hybrid_clock.cc:648] HybridClock initialized: now 1790349841032986 us; error 48 us; skew 500 ppm
I20260925 15:24:01.033390 30050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:01 dist-test-slave-8vw7 krb5kdc[30037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:01.042883 30050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:01.044955 30050 webserver.cc:460] Webserver started at http://127.0.0.1:45007/ using document root <none> and password file <none>
I20260925 15:24:01.045799 30050 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:01.045940 30050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:01.046324 30050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:01.049333 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "8b824118a50a45909b49fec79fb0ed6d"
format_stamp: "Formatted at 2026-09-25 15:24:01 on dist-test-slave-8vw7"
server_key: "b53ca645e1fea5aa089df266d294f475"
server_key_iv: "11f2d4ce83330e2f627705359bbabd81"
server_key_version: "encryptionkey@0"
I20260925 15:24:01.050033 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "8b824118a50a45909b49fec79fb0ed6d"
format_stamp: "Formatted at 2026-09-25 15:24:01 on dist-test-slave-8vw7"
server_key: "b53ca645e1fea5aa089df266d294f475"
server_key_iv: "11f2d4ce83330e2f627705359bbabd81"
server_key_version: "encryptionkey@0"
I20260925 15:24:01.054709 30050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:24:01.057879 30065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:01.059073 30050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:01.059239 30050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "8b824118a50a45909b49fec79fb0ed6d"
format_stamp: "Formatted at 2026-09-25 15:24:01 on dist-test-slave-8vw7"
server_key: "b53ca645e1fea5aa089df266d294f475"
server_key_iv: "11f2d4ce83330e2f627705359bbabd81"
server_key_version: "encryptionkey@0"
I20260925 15:24:01.059415 30050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:01.064803 30050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:01.068436 30050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:01.068709 30050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:01.086293 30050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39729
I20260925 15:24:01.086369 30116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39729 every 8 connection(s)
I20260925 15:24:01.087599 30050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:01.090994 30117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:01.093323 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30050
I20260925 15:24:01.093494 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:01.093937 9097 external_mini_cluster.cc:1428] Setting key 9f168c6fcbd48f8022b7d84cf8bede5f
I20260925 15:24:01.102542 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:01.095699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47961 (local address 127.0.0.1:39729)
0925 15:24:01.096510 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:01.096530 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:01.096579 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:24:01.098849 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:01.098875 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:01.099307 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:01.099735 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:01.100671 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:01.100685 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:01.100701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:01.100718 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:01.100796 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:01.100827 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:01.101067 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:01.101203 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:24:01.101391 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260925 15:24:01.103804 30117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d: Bootstrap starting.
I20260925 15:24:01.106001 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:01.107048 30117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:01.108197 30117 log.cc:824] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:01.111378 30117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d: No bootstrap required, opened a new log
I20260925 15:24:01.113725 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:01.111116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37079 (local address 127.0.0.1:39729)
0925 15:24:01.111284 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:01.111291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:01.111841 (+ 550us) server_negotiation.cc:407] Connection header received
0925 15:24:01.111998 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:01.112001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:01.112042 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:01.112115 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:01.113060 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:01.113064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:01.113067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:01.113081 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:01.113119 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:01.113133 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:01.113228 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:01.113361 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:24:01.113401 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:24:01.119026 30117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b824118a50a45909b49fec79fb0ed6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260925 15:24:01.119361 30117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:01.119483 30117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b824118a50a45909b49fec79fb0ed6d, State: Initialized, Role: FOLLOWER
I20260925 15:24:01.119988 30117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b824118a50a45909b49fec79fb0ed6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260925 15:24:01.120213 30117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:01.120345 30117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:01.120515 30117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:01.122696 30117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b824118a50a45909b49fec79fb0ed6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260925 15:24:01.123148 30117 leader_election.cc:302] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b824118a50a45909b49fec79fb0ed6d; no voters:
I20260925 15:24:01.123631 30117 leader_election.cc:288] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:01.123732 30127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:01.124418 30127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [term 1 LEADER]: Becoming Leader. State: Replica: 8b824118a50a45909b49fec79fb0ed6d, State: Running, Role: LEADER
I20260925 15:24:01.125005 30127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b824118a50a45909b49fec79fb0ed6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260925 15:24:01.125751 30117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:01.127959 30129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b824118a50a45909b49fec79fb0ed6d. Latest consensus state: current_term: 1 leader_uuid: "8b824118a50a45909b49fec79fb0ed6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b824118a50a45909b49fec79fb0ed6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260925 15:24:01.127933 30128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b824118a50a45909b49fec79fb0ed6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b824118a50a45909b49fec79fb0ed6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260925 15:24:01.128407 30129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:01.128403 30128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:01.130223 30133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:01.134282 30133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:01.148322 30133 catalog_manager.cc:1347] Generated new cluster ID: 1f101d8460b64b32a81b9388bccd1663
I20260925 15:24:01.148478 30133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:01.169334 30133 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:01.169914 30133 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:01.175729 30133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b824118a50a45909b49fec79fb0ed6d: Generated new TSK 0
I20260925 15:24:01.176479 30133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:01.181811 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30050
W20260925 15:24:01.182835 30112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47961
W20260925 15:24:01.183470 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30050: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:01.183609 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30050
2026-09-25T15:24:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1807 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:01 dist-test-slave-8vw7 krb5kdc[30148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:01 dist-test-slave-8vw7 krb5kdc[30148](info): set up 2 sockets
Sep 25 15:24:01 dist-test-slave-8vw7 krb5kdc[30148](info): commencing operation
krb5kdc: starting...
W20260925 15:24:02.253602 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:02 dist-test-slave-8vw7 krb5kdc[30148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:02.492118 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:02.813016 30161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:02.813302 30161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:02.813393 30161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:02.820350 30161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:02.820442 30161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:02.820485 30161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:02.820534 30161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:02.820580 30161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:02.829531 30161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:02.830637 30161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:02.831789 30161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:02.842976 30166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:02.843022 30169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:02.843057 30167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:02.844081 30161 server_base.cc:1034] running on GCE node
I20260925 15:24:02.844866 30161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:02.846227 30161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:02.847447 30161 hybrid_clock.cc:648] HybridClock initialized: now 1790349842847403 us; error 32 us; skew 500 ppm
I20260925 15:24:02.847807 30161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:02 dist-test-slave-8vw7 krb5kdc[30148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:02.850939 30161 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:02.852362 30161 webserver.cc:460] Webserver started at http://127.0.0.1:44461/ using document root <none> and password file <none>
I20260925 15:24:02.853051 30161 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:02.853153 30161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:02.853406 30161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:02.856186 30161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0"
format_stamp: "Formatted at 2026-09-25 15:24:02 on dist-test-slave-8vw7"
server_key: "00a3ed4e55d5ee507bb2d08cb3b885d2"
server_key_iv: "89447478cee8defc83549bc3ded41375"
server_key_version: "encryptionkey@0"
I20260925 15:24:02.856906 30161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0"
format_stamp: "Formatted at 2026-09-25 15:24:02 on dist-test-slave-8vw7"
server_key: "00a3ed4e55d5ee507bb2d08cb3b885d2"
server_key_iv: "89447478cee8defc83549bc3ded41375"
server_key_version: "encryptionkey@0"
I20260925 15:24:02.861898 30161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:24:02.865108 30176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:02.866360 30161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:24:02.866529 30161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0"
format_stamp: "Formatted at 2026-09-25 15:24:02 on dist-test-slave-8vw7"
server_key: "00a3ed4e55d5ee507bb2d08cb3b885d2"
server_key_iv: "89447478cee8defc83549bc3ded41375"
server_key_version: "encryptionkey@0"
I20260925 15:24:02.866736 30161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:02.882063 30161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:02.885787 30161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:02.886063 30161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:02.903956 30161 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36689
I20260925 15:24:02.903986 30227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36689 every 8 connection(s)
I20260925 15:24:02.905262 30161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:02.908613 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30161
I20260925 15:24:02.908816 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:02.909001 30228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:02.909415 9097 external_mini_cluster.cc:1428] Setting key 2a89c7647fffc47a5198faa69992aff8
I20260925 15:24:02.922972 30228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0: Bootstrap starting.
Sep 25 15:24:02 dist-test-slave-8vw7 krb5kdc[30148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:24:02.926136 30228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:02.927167 30228 log.cc:824] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:02.929842 30228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0: No bootstrap required, opened a new log
I20260925 15:24:02.931488 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:02.911106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:53537 (local address 127.0.0.1:36689)
0925 15:24:02.911892 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:02.911905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:02.911944 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:24:02.913573 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:02.913592 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:02.913957 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:02.914263 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:02.916044 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:02.917416 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:02.919186 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:02.919879 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:02.920012 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:02.924583 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:02.924590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:02.924603 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:02.924639 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:02.927648 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:02.928380 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:02.928386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:02.928392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:02.928538 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:02.929044 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:02.929051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:02.929054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:02.929498 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:02.929847 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:02.930328 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 15:24:02.930611 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":364,"threads_started":1}
I20260925 15:24:02.935333 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:02.938194 30228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260925 15:24:02.938539 30228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:02.938652 30228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4a9121cc3845f7bdf596dc5eb71cd0, State: Initialized, Role: FOLLOWER
I20260925 15:24:02.939224 30228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260925 15:24:02.939493 30228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:02.939651 30228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:02.939836 30228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:02.941933 30228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260925 15:24:02.942431 30228 leader_election.cc:302] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d4a9121cc3845f7bdf596dc5eb71cd0; no voters:
I20260925 15:24:02.943044 30228 leader_election.cc:288] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:02.943152 30238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:02.944736 30238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d4a9121cc3845f7bdf596dc5eb71cd0, State: Running, Role: LEADER
I20260925 15:24:02.945200 30228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:02.945400 30238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260925 15:24:02.949455 30239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } } }
I20260925 15:24:02.949887 30239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:02.952672 30248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:02.953235 30240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d4a9121cc3845f7bdf596dc5eb71cd0. Latest consensus state: current_term: 1 leader_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d4a9121cc3845f7bdf596dc5eb71cd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } } }
I20260925 15:24:02.953555 30240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:02.956599 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:02.939629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:38923 (local address 127.0.0.1:36689)
0925 15:24:02.939853 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:02.939861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:02.940340 (+ 479us) server_negotiation.cc:407] Connection header received
0925 15:24:02.940519 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:02.940527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:02.940598 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:02.940701 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:02.942063 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:02.942899 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:02.944518 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:02.945203 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:02.945348 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:02.948548 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:02.948554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:02.948560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:02.948595 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:02.953297 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:02.954072 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:02.954077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:02.954080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:02.954247 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:02.954824 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:02.954830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:02.954833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:02.955186 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:02.955499 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:02.955852 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 15:24:02.956029 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:24:02.957950 30248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:02.966706 30248 catalog_manager.cc:1347] Generated new cluster ID: a166eb32f25d420f949daa2a5caac893
I20260925 15:24:02.966806 30248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:02.995051 30248 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:02.996840 30248 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:03.003211 30248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d4a9121cc3845f7bdf596dc5eb71cd0: Generated new TSK 0
I20260925 15:24:03.004238 30248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:03.012697 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30161
I20260925 15:24:03.014029 30223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:03.107499 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30161
W20260925 15:24:03.124101 30146 connection.cc:535] client connection to 127.0.0.1:36689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36689): Connection reset by peer (error 104)
2026-09-25T15:24:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:03 dist-test-slave-8vw7 krb5kdc[30260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:03 dist-test-slave-8vw7 krb5kdc[30260](info): set up 2 sockets
Sep 25 15:24:03 dist-test-slave-8vw7 krb5kdc[30260](info): commencing operation
krb5kdc: starting...
W20260925 15:24:04.193552 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:04 dist-test-slave-8vw7 krb5kdc[30260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:04.432302 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:04.752545 30273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:04.752813 30273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:04.752928 30273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:04.759864 30273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:04.759963 30273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:04.760071 30273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:04.760185 30273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:04.760268 30273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:04.769649 30273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:04.770833 30273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:04.772051 30273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:04.783648 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:04.783699 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:04.783661 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:04.784817 30273 server_base.cc:1034] running on GCE node
I20260925 15:24:04.785674 30273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:04.787148 30273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:04.788369 30273 hybrid_clock.cc:648] HybridClock initialized: now 1790349844788332 us; error 48 us; skew 500 ppm
I20260925 15:24:04.788709 30273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:04 dist-test-slave-8vw7 krb5kdc[30260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:04.791659 30273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:04.793146 30273 webserver.cc:460] Webserver started at http://127.0.0.1:44495/ using document root <none> and password file <none>
I20260925 15:24:04.793787 30273 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:04.793927 30273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:04.794231 30273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:04.797106 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "079f830f37f64fe9a5db978b2524c15e"
format_stamp: "Formatted at 2026-09-25 15:24:04 on dist-test-slave-8vw7"
server_key: "1bde5e48cb75a1141be5005f2b8cdfc2"
server_key_iv: "4505e9ba78e5fb9efc9af5170b7ccaf3"
server_key_version: "encryptionkey@0"
I20260925 15:24:04.797864 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "079f830f37f64fe9a5db978b2524c15e"
format_stamp: "Formatted at 2026-09-25 15:24:04 on dist-test-slave-8vw7"
server_key: "1bde5e48cb75a1141be5005f2b8cdfc2"
server_key_iv: "4505e9ba78e5fb9efc9af5170b7ccaf3"
server_key_version: "encryptionkey@0"
I20260925 15:24:04.802712 30273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:24:04.806061 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:04.807325 30273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:24:04.807492 30273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "079f830f37f64fe9a5db978b2524c15e"
format_stamp: "Formatted at 2026-09-25 15:24:04 on dist-test-slave-8vw7"
server_key: "1bde5e48cb75a1141be5005f2b8cdfc2"
server_key_iv: "4505e9ba78e5fb9efc9af5170b7ccaf3"
server_key_version: "encryptionkey@0"
I20260925 15:24:04.807700 30273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:04.820490 30273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:04.824163 30273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:04.824453 30273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:04.842275 30273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44983
I20260925 15:24:04.842351 30339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44983 every 8 connection(s)
I20260925 15:24:04.843626 30273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:04.844691 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30273
I20260925 15:24:04.844887 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:04.845359 9097 external_mini_cluster.cc:1428] Setting key 31f47462e15f8b3e31cf2a7501a6f5e8
I20260925 15:24:04.847712 30340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:04.858394 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:04.847037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:59205 (local address 127.0.0.1:44983)
0925 15:24:04.847778 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:04.847790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:04.847852 (+ 62us) server_negotiation.cc:407] Connection header received
0925 15:24:04.849603 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:04.849626 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:04.850005 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:04.850331 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:04.850907 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:04.852237 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:04.854154 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:04.854781 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:04.854900 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:04.856174 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:04.856181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:04.856193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:04.856739 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:04.856833 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:04.856862 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:04.857061 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:04.857317 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 15:24:04.857500 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":344,"threads_started":1}
I20260925 15:24:04.859819 30340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e: Bootstrap starting.
I20260925 15:24:04.862208 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:04.863047 30340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:04.863994 30340 log.cc:824] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:04.867080 30340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e: No bootstrap required, opened a new log
I20260925 15:24:04.872514 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:04.866572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:58021 (local address 127.0.0.1:44983)
0925 15:24:04.866733 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:04.866741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:04.867112 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:24:04.867342 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:04.867346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:04.867399 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:04.867512 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:04.868002 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:04.868767 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:04.870107 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:04.870533 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:04.870682 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:04.871778 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:04.871782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:04.871785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:04.871802 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:04.871835 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:04.871851 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:04.871976 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:04.872212 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:24:04.872264 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:24:04.874957 30340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079f830f37f64fe9a5db978b2524c15e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260925 15:24:04.875316 30340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:04.875418 30340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079f830f37f64fe9a5db978b2524c15e, State: Initialized, Role: FOLLOWER
I20260925 15:24:04.875933 30340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079f830f37f64fe9a5db978b2524c15e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260925 15:24:04.876202 30340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:04.876344 30340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:04.876518 30340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:04.878975 30340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079f830f37f64fe9a5db978b2524c15e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260925 15:24:04.879434 30340 leader_election.cc:302] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 079f830f37f64fe9a5db978b2524c15e; no voters:
I20260925 15:24:04.879944 30340 leader_election.cc:288] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:04.880059 30350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:04.880730 30350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [term 1 LEADER]: Becoming Leader. State: Replica: 079f830f37f64fe9a5db978b2524c15e, State: Running, Role: LEADER
I20260925 15:24:04.881276 30350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079f830f37f64fe9a5db978b2524c15e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } }
I20260925 15:24:04.881946 30340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:04.883962 30351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "079f830f37f64fe9a5db978b2524c15e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079f830f37f64fe9a5db978b2524c15e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } } }
I20260925 15:24:04.884270 30351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:04.886179 30352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 079f830f37f64fe9a5db978b2524c15e. Latest consensus state: current_term: 1 leader_uuid: "079f830f37f64fe9a5db978b2524c15e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079f830f37f64fe9a5db978b2524c15e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44983 } } }
I20260925 15:24:04.886547 30352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:04.890473 30356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:04.896278 30356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:04.904337 30356 catalog_manager.cc:1347] Generated new cluster ID: 7435d91415a8435c8c6c0feeb367eef7
I20260925 15:24:04.904438 30356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:04.914978 30356 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:04.916167 30356 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:04.923075 30356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 079f830f37f64fe9a5db978b2524c15e: Generated new TSK 0
I20260925 15:24:04.923801 30356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:04.936360 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30273
W20260925 15:24:04.938088 30335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.231:59205
W20260925 15:24:04.938808 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30273: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:04.938959 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30273
W20260925 15:24:04.955129 30258 connection.cc:535] client connection to 127.0.0.1:44983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44983): Connection reset by peer (error 104)
2026-09-25T15:24:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:05 dist-test-slave-8vw7 krb5kdc[30371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:05 dist-test-slave-8vw7 krb5kdc[30371](info): set up 2 sockets
Sep 25 15:24:05 dist-test-slave-8vw7 krb5kdc[30371](info): commencing operation
krb5kdc: starting...
W20260925 15:24:06.021396 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:06 dist-test-slave-8vw7 krb5kdc[30371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:06.246204 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:06.560717 30384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:06.561026 30384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:06.561193 30384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:06.568102 30384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:06.568195 30384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:06.568306 30384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:06.568427 30384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:06.568509 30384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:06.577862 30384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:06.578981 30384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:06.580216 30384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:06.591750 30389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:06.591701 30390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:06.591791 30392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:06.592782 30384 server_base.cc:1034] running on GCE node
I20260925 15:24:06.593508 30384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:06.594776 30384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:06.595968 30384 hybrid_clock.cc:648] HybridClock initialized: now 1790349846595928 us; error 28 us; skew 500 ppm
I20260925 15:24:06.596292 30384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:06 dist-test-slave-8vw7 krb5kdc[30371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:06.607293 30384 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:06.608623 30384 webserver.cc:460] Webserver started at http://127.0.0.1:46747/ using document root <none> and password file <none>
I20260925 15:24:06.609328 30384 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:06.609427 30384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:06.609691 30384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:06.612455 30384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "39c642f30de64784a74669b4ce3c5f8f"
format_stamp: "Formatted at 2026-09-25 15:24:06 on dist-test-slave-8vw7"
server_key: "5d279ab988a4029cc05612e97545ef2f"
server_key_iv: "7aeac237a74abfc3b1d6b3eb73332152"
server_key_version: "encryptionkey@0"
I20260925 15:24:06.613221 30384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "39c642f30de64784a74669b4ce3c5f8f"
format_stamp: "Formatted at 2026-09-25 15:24:06 on dist-test-slave-8vw7"
server_key: "5d279ab988a4029cc05612e97545ef2f"
server_key_iv: "7aeac237a74abfc3b1d6b3eb73332152"
server_key_version: "encryptionkey@0"
I20260925 15:24:06.617998 30384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:24:06.621308 30399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:06.622550 30384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:24:06.622725 30384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "39c642f30de64784a74669b4ce3c5f8f"
format_stamp: "Formatted at 2026-09-25 15:24:06 on dist-test-slave-8vw7"
server_key: "5d279ab988a4029cc05612e97545ef2f"
server_key_iv: "7aeac237a74abfc3b1d6b3eb73332152"
server_key_version: "encryptionkey@0"
I20260925 15:24:06.622906 30384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:06.628224 30384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:06.631853 30384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:06.632123 30384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:06.649665 30384 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35777
I20260925 15:24:06.649715 30450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35777 every 8 connection(s)
I20260925 15:24:06.650946 30384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:06.654333 30451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:06.656834 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30384
I20260925 15:24:06.657049 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:06.657487 9097 external_mini_cluster.cc:1428] Setting key 770db093a28e28b6ea7c38c35f6fc505
I20260925 15:24:06.665306 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:06.659371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:33233 (local address 127.0.0.1:35777)
0925 15:24:06.660163 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:06.660174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:06.660215 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:24:06.661986 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:06.662011 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:06.662351 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:06.662694 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:06.663731 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:06.663737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:06.663748 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:06.663765 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:06.663826 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:06.663846 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:06.664011 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:06.664140 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:24:06.664339 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":356,"threads_started":1}
I20260925 15:24:06.665932 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f: Bootstrap starting.
I20260925 15:24:06.668817 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:06.669194 30451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:06.670176 30451 log.cc:824] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:06.673226 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f: No bootstrap required, opened a new log
I20260925 15:24:06.679966 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:06.673102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:52703 (local address 127.0.0.1:35777)
0925 15:24:06.673299 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:06.673308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:06.677980 (+ 4672us) server_negotiation.cc:407] Connection header received
0925 15:24:06.678178 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:06.678183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:06.678248 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:06.678359 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:06.679319 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:06.679325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:06.679330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:06.679348 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:06.679394 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:06.679413 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:06.679557 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:06.679639 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 15:24:06.679709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:24:06.680652 30451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c642f30de64784a74669b4ce3c5f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260925 15:24:06.681044 30451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:06.681162 30451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c642f30de64784a74669b4ce3c5f8f, State: Initialized, Role: FOLLOWER
I20260925 15:24:06.681686 30451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c642f30de64784a74669b4ce3c5f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260925 15:24:06.681929 30451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:06.682065 30451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:06.682242 30451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:06.684337 30451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c642f30de64784a74669b4ce3c5f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260925 15:24:06.684818 30451 leader_election.cc:302] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c642f30de64784a74669b4ce3c5f8f; no voters:
I20260925 15:24:06.685400 30451 leader_election.cc:288] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:06.685513 30461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:06.686848 30461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [term 1 LEADER]: Becoming Leader. State: Replica: 39c642f30de64784a74669b4ce3c5f8f, State: Running, Role: LEADER
I20260925 15:24:06.687446 30451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:06.687319 30461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c642f30de64784a74669b4ce3c5f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260925 15:24:06.690711 30462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c642f30de64784a74669b4ce3c5f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c642f30de64784a74669b4ce3c5f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } } }
I20260925 15:24:06.691146 30462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:06.690686 30463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c642f30de64784a74669b4ce3c5f8f. Latest consensus state: current_term: 1 leader_uuid: "39c642f30de64784a74669b4ce3c5f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c642f30de64784a74669b4ce3c5f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } } }
I20260925 15:24:06.692452 30463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:06.698735 30469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:06.704548 30469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:06.713910 30469 catalog_manager.cc:1347] Generated new cluster ID: 7ae7813bf1c549b18927cee47b76a1c3
I20260925 15:24:06.714021 30469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:06.731403 30469 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:06.731837 30469 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:06.741595 30469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c642f30de64784a74669b4ce3c5f8f: Generated new TSK 0
I20260925 15:24:06.742571 30469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:06.746567 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30384
W20260925 15:24:06.747604 30446 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.231:33233
W20260925 15:24:06.748260 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30384: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:06.748412 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30384
2026-09-25T15:24:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1809 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:06 dist-test-slave-8vw7 krb5kdc[30482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:06 dist-test-slave-8vw7 krb5kdc[30482](info): set up 2 sockets
Sep 25 15:24:06 dist-test-slave-8vw7 krb5kdc[30482](info): commencing operation
krb5kdc: starting...
W20260925 15:24:07.833654 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:07 dist-test-slave-8vw7 krb5kdc[30482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:08.075860 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:08.392679 30495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:08.393046 30495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:08.393206 30495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:08.400323 30495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:08.400467 30495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:08.400578 30495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:08.400688 30495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:08.400776 30495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:08.410593 30495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:08.411883 30495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:08.413257 30495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:08.425206 30500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:08.425271 30503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:08.425330 30501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:08.425949 30495 server_base.cc:1034] running on GCE node
I20260925 15:24:08.426693 30495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:08.427994 30495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:08.429214 30495 hybrid_clock.cc:648] HybridClock initialized: now 1790349848429173 us; error 46 us; skew 500 ppm
I20260925 15:24:08.429548 30495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:08 dist-test-slave-8vw7 krb5kdc[30482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:08.432586 30495 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:08.433984 30495 webserver.cc:460] Webserver started at http://127.0.0.1:39221/ using document root <none> and password file <none>
I20260925 15:24:08.434639 30495 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:08.434783 30495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:08.435093 30495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:08.438043 30495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "ccee21eca40d4fd3bc81aed488576db7"
format_stamp: "Formatted at 2026-09-25 15:24:08 on dist-test-slave-8vw7"
server_key: "14093c1d5692d54605d711c56f52944e"
server_key_iv: "cf35ee4c1ce4f19b4cd82ecdf4cd3c08"
server_key_version: "encryptionkey@0"
I20260925 15:24:08.438776 30495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "ccee21eca40d4fd3bc81aed488576db7"
format_stamp: "Formatted at 2026-09-25 15:24:08 on dist-test-slave-8vw7"
server_key: "14093c1d5692d54605d711c56f52944e"
server_key_iv: "cf35ee4c1ce4f19b4cd82ecdf4cd3c08"
server_key_version: "encryptionkey@0"
I20260925 15:24:08.443617 30495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:24:08.446926 30510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:08.448149 30495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:08.448338 30495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "ccee21eca40d4fd3bc81aed488576db7"
format_stamp: "Formatted at 2026-09-25 15:24:08 on dist-test-slave-8vw7"
server_key: "14093c1d5692d54605d711c56f52944e"
server_key_iv: "cf35ee4c1ce4f19b4cd82ecdf4cd3c08"
server_key_version: "encryptionkey@0"
I20260925 15:24:08.448534 30495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:08.462518 30495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:08.466641 30495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:08.466917 30495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:08.484347 30495 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41963
I20260925 15:24:08.484378 30561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41963 every 8 connection(s)
I20260925 15:24:08.485626 30495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:08.487317 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30495
I20260925 15:24:08.487488 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:08.488021 9097 external_mini_cluster.cc:1428] Setting key 3e2316377cb8ff6c2ffd3bef4578be64
I20260925 15:24:08.490015 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:24:08 dist-test-slave-8vw7 krb5kdc[30482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:24:08.502619 30562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7: Bootstrap starting.
I20260925 15:24:08.506494 30562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:08.507673 30562 log.cc:824] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:08.510926 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:08.489738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:45111 (local address 127.0.0.1:41963)
0925 15:24:08.490468 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:08.490479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:08.490515 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:24:08.492009 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:08.492027 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:08.492387 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:08.492701 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:08.494370 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:08.495659 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:08.497396 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:08.498031 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:08.498153 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:08.502660 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:08.502671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:08.502687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:08.502734 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:08.506346 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:08.507170 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:08.507178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:08.507185 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:08.507394 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:08.507840 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:08.507845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:08.507848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:08.508463 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:08.508736 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:08.509392 (+ 656us) server_negotiation.cc:299] Negotiation successful
0925 15:24:08.509696 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":349,"threads_started":1}
I20260925 15:24:08.510974 30562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7: No bootstrap required, opened a new log
I20260925 15:24:08.514487 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:08.518919 30562 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccee21eca40d4fd3bc81aed488576db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260925 15:24:08.519272 30562 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:08.519381 30562 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccee21eca40d4fd3bc81aed488576db7, State: Initialized, Role: FOLLOWER
I20260925 15:24:08.519917 30562 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccee21eca40d4fd3bc81aed488576db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260925 15:24:08.520126 30562 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:08.520273 30562 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:08.520424 30562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:08.522518 30562 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccee21eca40d4fd3bc81aed488576db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260925 15:24:08.522980 30562 leader_election.cc:302] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccee21eca40d4fd3bc81aed488576db7; no voters:
I20260925 15:24:08.523569 30562 leader_election.cc:288] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:08.523676 30572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:08.525184 30572 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [term 1 LEADER]: Becoming Leader. State: Replica: ccee21eca40d4fd3bc81aed488576db7, State: Running, Role: LEADER
I20260925 15:24:08.525705 30572 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccee21eca40d4fd3bc81aed488576db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260925 15:24:08.525894 30562 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:08.529032 30574 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccee21eca40d4fd3bc81aed488576db7. Latest consensus state: current_term: 1 leader_uuid: "ccee21eca40d4fd3bc81aed488576db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccee21eca40d4fd3bc81aed488576db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } } }
I20260925 15:24:08.529469 30574 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:08.530138 30573 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccee21eca40d4fd3bc81aed488576db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccee21eca40d4fd3bc81aed488576db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } } }
I20260925 15:24:08.530426 30573 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:08.532692 30578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:08.535023 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:08.518755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:36323 (local address 127.0.0.1:41963)
0925 15:24:08.518909 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:08.518917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:08.519354 (+ 437us) server_negotiation.cc:407] Connection header received
0925 15:24:08.519606 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:08.519610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:08.519709 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:08.519820 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:08.521411 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:08.522161 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:08.523468 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:08.524169 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:08.524298 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:08.526957 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:08.526965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:08.526972 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:08.527013 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:08.531850 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:08.532796 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:08.532802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:08.532806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:08.533034 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:08.533606 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:08.533609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:08.533611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:08.533874 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:08.534159 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:08.534476 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:24:08.534637 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260925 15:24:08.540102 30578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:08.550336 30578 catalog_manager.cc:1347] Generated new cluster ID: 5fb8731bf3984092adf28ec69ea2980e
I20260925 15:24:08.550434 30578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:08.570089 30578 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:08.571282 30578 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:08.584492 30578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccee21eca40d4fd3bc81aed488576db7: Generated new TSK 0
I20260925 15:24:08.585316 30578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:08.594895 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30495
I20260925 15:24:08.595865 30557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:08.687052 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30495
W20260925 15:24:08.703761 30480 connection.cc:535] client connection to 127.0.0.1:41963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41963): Connection reset by peer (error 104)
2026-09-25T15:24:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:08 dist-test-slave-8vw7 krb5kdc[30594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:08 dist-test-slave-8vw7 krb5kdc[30594](info): set up 2 sockets
Sep 25 15:24:08 dist-test-slave-8vw7 krb5kdc[30594](info): commencing operation
krb5kdc: starting...
W20260925 15:24:09.773295 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:09 dist-test-slave-8vw7 krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:10.024209 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:10.344990 30607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:10.345263 30607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:10.345453 30607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:10.352519 30607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:10.352655 30607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:10.352751 30607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:10.352859 30607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:10.352948 30607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:10.362425 30607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:10.363624 30607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:10.364876 30607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:10.376477 30613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:10.376907 30615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:10.377035 30612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:10.377503 30607 server_base.cc:1034] running on GCE node
I20260925 15:24:10.378298 30607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:10.379596 30607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:10.380829 30607 hybrid_clock.cc:648] HybridClock initialized: now 1790349850380793 us; error 47 us; skew 500 ppm
I20260925 15:24:10.381208 30607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:10 dist-test-slave-8vw7 krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:10.384244 30607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:10.385809 30607 webserver.cc:460] Webserver started at http://127.0.0.1:45689/ using document root <none> and password file <none>
I20260925 15:24:10.386444 30607 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:10.386590 30607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:10.386880 30607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:10.389780 30607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "9cdd0e245d0040f38f9837091c29cbdf"
format_stamp: "Formatted at 2026-09-25 15:24:10 on dist-test-slave-8vw7"
server_key: "39666d50d667b7154f20c8f28605e10e"
server_key_iv: "045707fa2b7c183572295fe693c24b84"
server_key_version: "encryptionkey@0"
I20260925 15:24:10.390516 30607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "9cdd0e245d0040f38f9837091c29cbdf"
format_stamp: "Formatted at 2026-09-25 15:24:10 on dist-test-slave-8vw7"
server_key: "39666d50d667b7154f20c8f28605e10e"
server_key_iv: "045707fa2b7c183572295fe693c24b84"
server_key_version: "encryptionkey@0"
I20260925 15:24:10.395341 30607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:24:10.398545 30622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:10.399699 30607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:24:10.399876 30607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "9cdd0e245d0040f38f9837091c29cbdf"
format_stamp: "Formatted at 2026-09-25 15:24:10 on dist-test-slave-8vw7"
server_key: "39666d50d667b7154f20c8f28605e10e"
server_key_iv: "045707fa2b7c183572295fe693c24b84"
server_key_version: "encryptionkey@0"
I20260925 15:24:10.400058 30607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:10.423126 30607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:10.427353 30607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:10.427652 30607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:10.445309 30607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44311
I20260925 15:24:10.445343 30673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44311 every 8 connection(s)
I20260925 15:24:10.446558 30607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:10.450059 30674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:10.456367 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30607
I20260925 15:24:10.456547 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:10.457057 9097 external_mini_cluster.cc:1428] Setting key 134c477afc4d9d3f650ae2d8ac2fcb24
I20260925 15:24:10.459934 30674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf: Bootstrap starting.
I20260925 15:24:10.463117 30674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:10.464183 30674 log.cc:824] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:10.466995 30674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf: No bootstrap required, opened a new log
I20260925 15:24:10.469817 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:10.458688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:38971 (local address 127.0.0.1:44311)
0925 15:24:10.459390 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:10.459400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:10.459437 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:24:10.461260 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:10.461282 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:10.461638 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:10.461942 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:10.462531 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:10.463876 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:10.465562 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:10.466198 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:10.466321 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:10.467606 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:10.467613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:10.467625 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:10.468131 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:10.468210 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:10.468233 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:10.468355 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:10.468666 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:24:10.468842 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":339,"threads_started":1}
I20260925 15:24:10.473299 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:10.475431 30674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdd0e245d0040f38f9837091c29cbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260925 15:24:10.475829 30674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:10.475941 30674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cdd0e245d0040f38f9837091c29cbdf, State: Initialized, Role: FOLLOWER
I20260925 15:24:10.476526 30674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdd0e245d0040f38f9837091c29cbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260925 15:24:10.476893 30674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:10.477072 30674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:10.477277 30674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:10.480278 30674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdd0e245d0040f38f9837091c29cbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260925 15:24:10.480732 30674 leader_election.cc:302] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cdd0e245d0040f38f9837091c29cbdf; no voters:
I20260925 15:24:10.481417 30674 leader_election.cc:288] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:10.481508 30684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:10.482743 30684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [term 1 LEADER]: Becoming Leader. State: Replica: 9cdd0e245d0040f38f9837091c29cbdf, State: Running, Role: LEADER
I20260925 15:24:10.483362 30684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdd0e245d0040f38f9837091c29cbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260925 15:24:10.483656 30674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:10.485277 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:10.478682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:50835 (local address 127.0.0.1:44311)
0925 15:24:10.478831 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:10.478835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:10.479210 (+ 375us) server_negotiation.cc:407] Connection header received
0925 15:24:10.479380 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:10.479383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:10.479448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:10.479514 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:10.480079 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:10.480825 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:10.482491 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:10.483169 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:10.483317 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:10.484441 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:10.484446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:10.484451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:10.484469 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:10.484531 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:10.484545 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:10.484666 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:10.484873 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:24:10.484920 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:24:10.490695 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cdd0e245d0040f38f9837091c29cbdf. Latest consensus state: current_term: 1 leader_uuid: "9cdd0e245d0040f38f9837091c29cbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdd0e245d0040f38f9837091c29cbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } } }
I20260925 15:24:10.490778 30685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cdd0e245d0040f38f9837091c29cbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cdd0e245d0040f38f9837091c29cbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } } }
I20260925 15:24:10.491192 30685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:10.491168 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:10.492851 30691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:10.498123 30691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:10.506354 30691 catalog_manager.cc:1347] Generated new cluster ID: f2e70d38da4e40969022dc78bd7ab212
I20260925 15:24:10.506456 30691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:10.518126 30691 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:10.519927 30691 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:10.530366 30691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cdd0e245d0040f38f9837091c29cbdf: Generated new TSK 0
I20260925 15:24:10.531255 30691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:10.537262 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30607
W20260925 15:24:10.538308 30669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.231:38971
W20260925 15:24:10.538996 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:10.539144 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30607
W20260925 15:24:10.552222 30592 connection.cc:535] client connection to 127.0.0.1:44311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44311): Connection reset by peer (error 104)
2026-09-25T15:24:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:10 dist-test-slave-8vw7 krb5kdc[30705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:10 dist-test-slave-8vw7 krb5kdc[30705](info): set up 2 sockets
Sep 25 15:24:10 dist-test-slave-8vw7 krb5kdc[30705](info): commencing operation
krb5kdc: starting...
W20260925 15:24:11.605372 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:11 dist-test-slave-8vw7 krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:11.835364 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:12.145256 30718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:12.145576 30718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:12.145741 30718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:12.152642 30718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:12.152793 30718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:12.152886 30718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:12.153007 30718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:12.153085 30718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:12.162551 30718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:12.163775 30718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:12.165064 30718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:12.176230 30723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:12.176362 30726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:12.176939 30724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:12.177616 30718 server_base.cc:1034] running on GCE node
I20260925 15:24:12.178311 30718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:12.179649 30718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:12.180909 30718 hybrid_clock.cc:648] HybridClock initialized: now 1790349852180868 us; error 50 us; skew 500 ppm
I20260925 15:24:12.181281 30718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:12 dist-test-slave-8vw7 krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:12.192931 30718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:12.194535 30718 webserver.cc:460] Webserver started at http://127.0.0.1:38909/ using document root <none> and password file <none>
I20260925 15:24:12.195187 30718 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:12.195328 30718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:12.195652 30718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:12.198559 30718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "f906a132bd874d5a916537c488fbae40"
format_stamp: "Formatted at 2026-09-25 15:24:12 on dist-test-slave-8vw7"
server_key: "423c3b2efa223ac02857b9268e4bc271"
server_key_iv: "d821c6ab2f64d5e24a7aa47e6a9b10c9"
server_key_version: "encryptionkey@0"
I20260925 15:24:12.199316 30718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "f906a132bd874d5a916537c488fbae40"
format_stamp: "Formatted at 2026-09-25 15:24:12 on dist-test-slave-8vw7"
server_key: "423c3b2efa223ac02857b9268e4bc271"
server_key_iv: "d821c6ab2f64d5e24a7aa47e6a9b10c9"
server_key_version: "encryptionkey@0"
I20260925 15:24:12.204067 30718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:24:12.207201 30733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:12.208364 30718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:12.208536 30718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "f906a132bd874d5a916537c488fbae40"
format_stamp: "Formatted at 2026-09-25 15:24:12 on dist-test-slave-8vw7"
server_key: "423c3b2efa223ac02857b9268e4bc271"
server_key_iv: "d821c6ab2f64d5e24a7aa47e6a9b10c9"
server_key_version: "encryptionkey@0"
I20260925 15:24:12.208736 30718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:12.213937 30718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:12.217552 30718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:12.217835 30718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:12.235608 30718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40745
I20260925 15:24:12.235656 30784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40745 every 8 connection(s)
I20260925 15:24:12.236937 30718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:12.237306 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30718
I20260925 15:24:12.237465 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:12.238039 9097 external_mini_cluster.cc:1428] Setting key 68161104d00810ea027d930ca461e85b
I20260925 15:24:12.241151 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:12.245450 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:12.239834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:35177 (local address 127.0.0.1:40745)
0925 15:24:12.240611 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:12.240623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:12.240666 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:24:12.242301 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:12.242322 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:12.242618 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:12.242934 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:12.243928 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:12.243933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:12.243944 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:12.243957 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:12.244031 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:12.244050 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:12.244221 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:12.244326 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:24:12.244515 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260925 15:24:12.249115 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:12.252550 30785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40: Bootstrap starting.
I20260925 15:24:12.255558 30785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:12.256122 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:12.253455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:52195 (local address 127.0.0.1:40745)
0925 15:24:12.253676 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:12.253685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:12.254226 (+ 541us) server_negotiation.cc:407] Connection header received
0925 15:24:12.254413 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:12.254418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:12.254474 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:12.254570 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:12.255493 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:12.255498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:12.255503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:12.255520 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:12.255568 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:12.255583 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:12.255694 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:12.255803 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:24:12.255868 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:24:12.256837 30785 log.cc:824] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:12.262451 30785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40: No bootstrap required, opened a new log
I20260925 15:24:12.269822 30785 raft_consensus.cc:348] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f906a132bd874d5a916537c488fbae40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 15:24:12.270208 30785 raft_consensus.cc:374] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:12.270284 30785 raft_consensus.cc:729] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f906a132bd874d5a916537c488fbae40, State: Initialized, Role: FOLLOWER
I20260925 15:24:12.270800 30785 consensus_queue.cc:260] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f906a132bd874d5a916537c488fbae40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 15:24:12.270993 30785 raft_consensus.cc:388] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:12.271085 30785 raft_consensus.cc:482] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:12.271204 30785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:12.273236 30785 raft_consensus.cc:504] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f906a132bd874d5a916537c488fbae40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 15:24:12.273639 30785 leader_election.cc:302] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f906a132bd874d5a916537c488fbae40; no voters:
I20260925 15:24:12.274089 30785 leader_election.cc:288] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:12.274230 30795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:12.274818 30795 raft_consensus.cc:686] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [term 1 LEADER]: Becoming Leader. State: Replica: f906a132bd874d5a916537c488fbae40, State: Running, Role: LEADER
I20260925 15:24:12.275287 30795 consensus_queue.cc:237] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f906a132bd874d5a916537c488fbae40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 15:24:12.275876 30785 sys_catalog.cc:564] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:12.277976 30796 sys_catalog.cc:455] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f906a132bd874d5a916537c488fbae40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f906a132bd874d5a916537c488fbae40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } } }
I20260925 15:24:12.278452 30796 sys_catalog.cc:458] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:12.279016 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f906a132bd874d5a916537c488fbae40. Latest consensus state: current_term: 1 leader_uuid: "f906a132bd874d5a916537c488fbae40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f906a132bd874d5a916537c488fbae40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } } }
I20260925 15:24:12.279356 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:12.280045 30802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:12.284652 30802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:12.292825 30802 catalog_manager.cc:1347] Generated new cluster ID: 4d2502177f4a4c57a16baa47790307e9
I20260925 15:24:12.292922 30802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:12.304735 30802 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:12.305229 30802 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:12.322428 30802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f906a132bd874d5a916537c488fbae40: Generated new TSK 0
I20260925 15:24:12.323331 30802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:12.329720 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30718
W20260925 15:24:12.330842 30780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.231:35177
W20260925 15:24:12.331547 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:12.331692 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30718
2026-09-25T15:24:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1795 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:12 dist-test-slave-8vw7 krb5kdc[30816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:12 dist-test-slave-8vw7 krb5kdc[30816](info): set up 2 sockets
Sep 25 15:24:12 dist-test-slave-8vw7 krb5kdc[30816](info): commencing operation
krb5kdc: starting...
W20260925 15:24:13.417480 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:13 dist-test-slave-8vw7 krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:13.654423 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34915
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:34915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:13.966054 30829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:13.966392 30829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:13.966568 30829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:13.973544 30829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:13.973687 30829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:13.973793 30829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:13.973903 30829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:13.973985 30829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:13.983402 30829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:34915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:34915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:13.984622 30829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:13.985885 30829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:13.997361 30834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:13.997400 30835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:13.997498 30837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:13.998409 30829 server_base.cc:1034] running on GCE node
I20260925 15:24:13.999359 30829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:14.000679 30829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:14.001868 30829 hybrid_clock.cc:648] HybridClock initialized: now 1790349854001830 us; error 27 us; skew 500 ppm
I20260925 15:24:14.002169 30829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:13 dist-test-slave-8vw7 krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349853, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:14.005208 30829 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:14.006608 30829 webserver.cc:460] Webserver started at http://127.8.226.126:38431/ using document root <none> and password file <none>
I20260925 15:24:14.007264 30829 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:14.007395 30829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:14.007700 30829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:14.010675 30829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "77314679e3ff4dad89aa0d71c2e41a19"
format_stamp: "Formatted at 2026-09-25 15:24:14 on dist-test-slave-8vw7"
server_key: "78d082361bab71bf89f4934f0677f3cc"
server_key_iv: "2684ec90136724e2e9bf818894945ced"
server_key_version: "encryptionkey@0"
I20260925 15:24:14.011425 30829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "77314679e3ff4dad89aa0d71c2e41a19"
format_stamp: "Formatted at 2026-09-25 15:24:14 on dist-test-slave-8vw7"
server_key: "78d082361bab71bf89f4934f0677f3cc"
server_key_iv: "2684ec90136724e2e9bf818894945ced"
server_key_version: "encryptionkey@0"
I20260925 15:24:14.016319 30829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:24:14.019670 30844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:14.020936 30829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:24:14.021176 30829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "77314679e3ff4dad89aa0d71c2e41a19"
format_stamp: "Formatted at 2026-09-25 15:24:14 on dist-test-slave-8vw7"
server_key: "78d082361bab71bf89f4934f0677f3cc"
server_key_iv: "2684ec90136724e2e9bf818894945ced"
server_key_version: "encryptionkey@0"
I20260925 15:24:14.021512 30829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:14.040795 30829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:14.044330 30829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:14.044588 30829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:14.061856 30829 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:34915
I20260925 15:24:14.061878 30895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:34915 every 8 connection(s)
I20260925 15:24:14.063097 30829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:14.066563 30896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:14.067260 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30829
I20260925 15:24:14.067435 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:14.068025 9097 external_mini_cluster.cc:1428] Setting key 52faa81c31815b95a3deb9652c5dd9e6
I20260925 15:24:14.079982 30896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19: Bootstrap starting.
Sep 25 15:24:14 dist-test-slave-8vw7 krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:24:14.083509 30896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:14.084641 30896 log.cc:824] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:14.087334 30896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19: No bootstrap required, opened a new log
I20260925 15:24:14.090982 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:14.069698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43723 (local address 127.8.226.126:34915)
0925 15:24:14.070439 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:14.070449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:14.070485 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:24:14.072516 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:14.072537 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:14.072895 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:14.073272 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:14.075064 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:14.076307 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:14.078091 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:14.078700 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:14.078818 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:14.083502 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:14.083509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:14.083521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:14.083556 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:14.086881 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:14.087645 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:14.087651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:14.087656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:14.087827 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:14.088281 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:14.088285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:14.088287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:14.088920 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:14.089326 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:14.089809 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 15:24:14.090093 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":353,"threads_started":1}
I20260925 15:24:14.094588 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:14.095677 30896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77314679e3ff4dad89aa0d71c2e41a19" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34915 } }
I20260925 15:24:14.096037 30896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:14.096138 30896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77314679e3ff4dad89aa0d71c2e41a19, State: Initialized, Role: FOLLOWER
I20260925 15:24:14.097183 30896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77314679e3ff4dad89aa0d71c2e41a19" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34915 } }
I20260925 15:24:14.097462 30896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:14.097600 30896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:14.097774 30896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:14.100678 30896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77314679e3ff4dad89aa0d71c2e41a19" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34915 } }
I20260925 15:24:14.101217 30896 leader_election.cc:302] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77314679e3ff4dad89aa0d71c2e41a19; no voters:
I20260925 15:24:14.101738 30896 leader_election.cc:288] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:14.101886 30906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:14.103790 30896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:14.104435 30906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [term 1 LEADER]: Becoming Leader. State: Replica: 77314679e3ff4dad89aa0d71c2e41a19, State: Running, Role: LEADER
I20260925 15:24:14.105023 30906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77314679e3ff4dad89aa0d71c2e41a19" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34915 } }
I20260925 15:24:14.108510 30907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77314679e3ff4dad89aa0d71c2e41a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77314679e3ff4dad89aa0d71c2e41a19" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34915 } } }
I20260925 15:24:14.108857 30907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:14.109277 30908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77314679e3ff4dad89aa0d71c2e41a19. Latest consensus state: current_term: 1 leader_uuid: "77314679e3ff4dad89aa0d71c2e41a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77314679e3ff4dad89aa0d71c2e41a19" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 34915 } } }
I20260925 15:24:14.109556 30908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:14.111923 30913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:14.117138 30913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:14.122116 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:14.099959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48925 (local address 127.8.226.126:34915)
0925 15:24:14.100124 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:14.100130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:14.100557 (+ 427us) server_negotiation.cc:407] Connection header received
0925 15:24:14.100679 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:14.100683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:14.100766 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:14.100870 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:14.102931 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:14.104058 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:14.106234 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:14.107194 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:14.107339 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:14.110895 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:14.110904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:14.110912 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:14.110958 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:14.116685 (+ 5727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:14.117973 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:14.117980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:14.117986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:14.118152 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:14.118904 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:14.118911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:14.118917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:14.119413 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:14.121021 (+ 1608us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:14.121274 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:24:14.121497 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:24:14.126375 30913 catalog_manager.cc:1347] Generated new cluster ID: ee4578bfe91b41e6a491d4ddcfbdb886
I20260925 15:24:14.126468 30913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:14.143234 30913 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:14.144843 30913 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:14.153992 30913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77314679e3ff4dad89aa0d71c2e41a19: Generated new TSK 0
I20260925 15:24:14.154793 30913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:14.165228 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30829
I20260925 15:24:14.166503 30891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:14.254956 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30829
W20260925 15:24:14.271463 30814 connection.cc:535] client connection to 127.8.226.126:34915 recv error: Network error: failed to read from TLS socket (remote: 127.8.226.126:34915): Connection reset by peer (error 104)
2026-09-25T15:24:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:14 dist-test-slave-8vw7 krb5kdc[30928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:14 dist-test-slave-8vw7 krb5kdc[30928](info): set up 2 sockets
Sep 25 15:24:14 dist-test-slave-8vw7 krb5kdc[30928](info): commencing operation
krb5kdc: starting...
W20260925 15:24:15.337451 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.011s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:15 dist-test-slave-8vw7 krb5kdc[30928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:15.563787 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39297
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:15.894157 30941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:15.894443 30941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:15.894534 30941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:15.901917 30941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:15.902035 30941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:15.902081 30941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:15.902133 30941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:15.902172 30941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:15.912129 30941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:39297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:15.913556 30941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:15.915000 30941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:15.927062 30946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:15.927114 30949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:15.927081 30947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:15.927791 30941 server_base.cc:1034] running on GCE node
I20260925 15:24:15.928661 30941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:15.930143 30941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:15.931366 30941 hybrid_clock.cc:648] HybridClock initialized: now 1790349855931313 us; error 48 us; skew 500 ppm
I20260925 15:24:15.931712 30941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:15 dist-test-slave-8vw7 krb5kdc[30928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:15.935173 30941 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:15.936656 30941 webserver.cc:460] Webserver started at http://127.8.226.126:32829/ using document root <none> and password file <none>
I20260925 15:24:15.937392 30941 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:15.937544 30941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:15.937853 30941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:15.940833 30941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "706a745b5d1a4dc9b5a88a69923ff8ea"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-8vw7"
server_key: "d0e24468b55d843402a8aeeb008ce563"
server_key_iv: "f966f312ddc36e6b39f832b2f1016353"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.941722 30941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "706a745b5d1a4dc9b5a88a69923ff8ea"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-8vw7"
server_key: "d0e24468b55d843402a8aeeb008ce563"
server_key_iv: "f966f312ddc36e6b39f832b2f1016353"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.947059 30941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:24:15.950856 30956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:15.952402 30941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:24:15.952617 30941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "706a745b5d1a4dc9b5a88a69923ff8ea"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-8vw7"
server_key: "d0e24468b55d843402a8aeeb008ce563"
server_key_iv: "f966f312ddc36e6b39f832b2f1016353"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.952808 30941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:15.970608 30941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:15.975098 30941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:15.975391 30941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:15.993880 30941 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:39297
I20260925 15:24:15.993899 31007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:39297 every 8 connection(s)
I20260925 15:24:15.995332 30941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:15.996268 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 30941
I20260925 15:24:15.996457 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:15.996984 9097 external_mini_cluster.cc:1428] Setting key fac86e429f77ae1e288284c12aa6cf49
I20260925 15:24:16.000128 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:16.009991 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:15.998895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57597 (local address 127.8.226.126:39297)
0925 15:24:15.999685 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:15.999697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:15.999740 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:24:16.001556 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.001579 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.001959 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.002363 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.002909 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.004201 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.005594 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.006259 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.006472 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.007739 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.007746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.007760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:16.008319 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.008392 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:16.008415 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.008603 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.008874 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.009063 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":346,"threads_started":1}
I20260925 15:24:16.010927 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea: Bootstrap starting.
I20260925 15:24:16.013813 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:16.014330 31008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:16.015338 31008 log.cc:824] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:16.018496 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea: No bootstrap required, opened a new log
I20260925 15:24:16.024860 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.018470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32819 (local address 127.8.226.126:39297)
0925 15:24:16.018695 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.018704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.019232 (+ 528us) server_negotiation.cc:407] Connection header received
0925 15:24:16.019442 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.019454 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.019512 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.019617 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.020086 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.020905 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.022247 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.022748 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.022890 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.024121 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.024126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.024129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:16.024149 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.024187 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:16.024202 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.024325 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.024523 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.024586 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:24:16.026759 31008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39297 } }
I20260925 15:24:16.027153 31008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:16.027261 31008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706a745b5d1a4dc9b5a88a69923ff8ea, State: Initialized, Role: FOLLOWER
I20260925 15:24:16.027868 31008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39297 } }
I20260925 15:24:16.028118 31008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:16.028270 31008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:16.028445 31008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:16.031013 31008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39297 } }
I20260925 15:24:16.031515 31008 leader_election.cc:302] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 706a745b5d1a4dc9b5a88a69923ff8ea; no voters:
I20260925 15:24:16.032094 31008 leader_election.cc:288] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:16.032243 31018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:16.033123 31018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [term 1 LEADER]: Becoming Leader. State: Replica: 706a745b5d1a4dc9b5a88a69923ff8ea, State: Running, Role: LEADER
I20260925 15:24:16.033687 31018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39297 } }
I20260925 15:24:16.034318 31008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:16.036505 31019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39297 } } }
I20260925 15:24:16.036942 31019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:16.038609 31020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 706a745b5d1a4dc9b5a88a69923ff8ea. Latest consensus state: current_term: 1 leader_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706a745b5d1a4dc9b5a88a69923ff8ea" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39297 } } }
I20260925 15:24:16.038882 31020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:16.039786 31026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:16.045193 31026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:16.053990 31026 catalog_manager.cc:1347] Generated new cluster ID: ca1044875fc8462da01fbd3dafc09148
I20260925 15:24:16.054301 31026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:16.098459 31026 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:16.099779 31026 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:16.104877 31026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 706a745b5d1a4dc9b5a88a69923ff8ea: Generated new TSK 0
I20260925 15:24:16.105916 31026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:16.120810 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30941
W20260925 15:24:16.122231 31003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57597
W20260925 15:24:16.123001 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 30941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:16.123162 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 30941
W20260925 15:24:16.140220 30926 connection.cc:535] client connection to 127.8.226.126:39297 recv error: Network error: failed to read from TLS socket (remote: 127.8.226.126:39297): Connection reset by peer (error 104)
2026-09-25T15:24:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:16 dist-test-slave-8vw7 krb5kdc[31039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:16 dist-test-slave-8vw7 krb5kdc[31039](info): set up 2 sockets
Sep 25 15:24:16 dist-test-slave-8vw7 krb5kdc[31039](info): commencing operation
krb5kdc: starting...
W20260925 15:24:17.225443 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:17 dist-test-slave-8vw7 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:17.461609 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39081
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:17.779822 31052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:17.780167 31052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:17.780321 31052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:17.788033 31052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:17.788173 31052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:17.788251 31052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:17.788357 31052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:17.788422 31052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:17.798187 31052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:39081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:17.799436 31052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:17.800734 31052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:17.812614 31057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:17.812590 31058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:17.812700 31060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:17.813254 31052 server_base.cc:1034] running on GCE node
I20260925 15:24:17.814000 31052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:17.815299 31052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:17.816517 31052 hybrid_clock.cc:648] HybridClock initialized: now 1790349857816473 us; error 57 us; skew 500 ppm
I20260925 15:24:17.816859 31052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:17 dist-test-slave-8vw7 krb5kdc[31039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349857, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:17.824312 31052 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:17.825780 31052 webserver.cc:460] Webserver started at http://127.8.226.126:44085/ using document root <none> and password file <none>
I20260925 15:24:17.826411 31052 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:17.826550 31052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:17.826853 31052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:17.829756 31052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40"
format_stamp: "Formatted at 2026-09-25 15:24:17 on dist-test-slave-8vw7"
server_key: "94d0d7745687db51def681ddca1ffa51"
server_key_iv: "0d354049b3b6f89419cf6f4dde30e129"
server_key_version: "encryptionkey@0"
I20260925 15:24:17.830483 31052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40"
format_stamp: "Formatted at 2026-09-25 15:24:17 on dist-test-slave-8vw7"
server_key: "94d0d7745687db51def681ddca1ffa51"
server_key_iv: "0d354049b3b6f89419cf6f4dde30e129"
server_key_version: "encryptionkey@0"
I20260925 15:24:17.835240 31052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:24:17.838492 31067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:17.839650 31052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:24:17.839835 31052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40"
format_stamp: "Formatted at 2026-09-25 15:24:17 on dist-test-slave-8vw7"
server_key: "94d0d7745687db51def681ddca1ffa51"
server_key_iv: "0d354049b3b6f89419cf6f4dde30e129"
server_key_version: "encryptionkey@0"
I20260925 15:24:17.840049 31052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:17.845517 31052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:17.849628 31052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:17.849913 31052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:17.867303 31052 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:39081
I20260925 15:24:17.867345 31118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:39081 every 8 connection(s)
I20260925 15:24:17.868582 31052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:17.871991 31119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:17.872552 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31052
I20260925 15:24:17.872740 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:17.873376 9097 external_mini_cluster.cc:1428] Setting key befafd5e7cadf17bf4dcabf7e035d07b
I20260925 15:24:17.881104 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.875033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55673 (local address 127.8.226.126:39081)
0925 15:24:17.875821 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.875832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.875875 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:24:17.877993 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.878013 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.878342 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.878650 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:17.879634 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:17.879640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:17.879652 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:17.879667 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:17.879723 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:17.879741 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:17.879893 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.880012 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.880204 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":375,"threads_started":1}
I20260925 15:24:17.884622 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:17.885243 31119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40: Bootstrap starting.
I20260925 15:24:17.888128 31119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:17.889256 31119 log.cc:824] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:17.891656 31119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40: No bootstrap required, opened a new log
I20260925 15:24:17.894218 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.888947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37967 (local address 127.8.226.126:39081)
0925 15:24:17.889130 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.889135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.892314 (+ 3179us) server_negotiation.cc:407] Connection header received
0925 15:24:17.892487 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.892491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.892536 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.892627 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:17.893578 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:17.893582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:17.893585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:17.893598 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:17.893636 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:17.893650 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:17.893741 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.893872 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.893912 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:24:17.899501 31119 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39081 } }
I20260925 15:24:17.899842 31119 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:17.899943 31119 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ecda2ba1fb47fb89192ff6a37b2e40, State: Initialized, Role: FOLLOWER
I20260925 15:24:17.900512 31119 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39081 } }
I20260925 15:24:17.900753 31119 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:17.900888 31119 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:17.901077 31119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:17.903298 31119 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39081 } }
I20260925 15:24:17.903746 31119 leader_election.cc:302] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ecda2ba1fb47fb89192ff6a37b2e40; no voters:
I20260925 15:24:17.904274 31119 leader_election.cc:288] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:17.904367 31129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:17.905068 31129 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [term 1 LEADER]: Becoming Leader. State: Replica: a4ecda2ba1fb47fb89192ff6a37b2e40, State: Running, Role: LEADER
I20260925 15:24:17.905565 31129 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39081 } }
I20260925 15:24:17.906183 31119 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:17.908272 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ecda2ba1fb47fb89192ff6a37b2e40. Latest consensus state: current_term: 1 leader_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39081 } } }
I20260925 15:24:17.909471 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:17.908668 31130 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecda2ba1fb47fb89192ff6a37b2e40" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39081 } } }
I20260925 15:24:17.909988 31130 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:17.911717 31138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:17.916298 31138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:17.924088 31138 catalog_manager.cc:1347] Generated new cluster ID: 2560fd13b5a54367928130bc3f406152
I20260925 15:24:17.924247 31138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:17.952126 31138 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:17.952546 31138 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:17.968979 31138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ecda2ba1fb47fb89192ff6a37b2e40: Generated new TSK 0
I20260925 15:24:17.969758 31138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:17.983031 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31052
W20260925 15:24:17.983948 31114 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55673
W20260925 15:24:17.984567 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31052: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:17.984719 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31052
2026-09-25T15:24:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:18 dist-test-slave-8vw7 krb5kdc[31150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:18 dist-test-slave-8vw7 krb5kdc[31150](info): set up 2 sockets
Sep 25 15:24:18 dist-test-slave-8vw7 krb5kdc[31150](info): commencing operation
krb5kdc: starting...
W20260925 15:24:19.065490 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:19 dist-test-slave-8vw7 krb5kdc[31150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:19.312476 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:40943
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:40943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:19.629703 31163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:19.630021 31163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:19.630180 31163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:19.637177 31163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:19.637310 31163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:19.637422 31163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:19.637524 31163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:19.637605 31163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:19.647718 31163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:40943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:40943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:19.649309 31163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:19.650470 31163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:19.661706 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:19.661737 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:19.661777 31171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:19.662528 31163 server_base.cc:1034] running on GCE node
I20260925 15:24:19.663187 31163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:19.664441 31163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:19.665645 31163 hybrid_clock.cc:648] HybridClock initialized: now 1790349859665580 us; error 49 us; skew 500 ppm
I20260925 15:24:19.665982 31163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:19 dist-test-slave-8vw7 krb5kdc[31150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:19.668921 31163 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:19.670298 31163 webserver.cc:460] Webserver started at http://127.8.226.126:41381/ using document root <none> and password file <none>
I20260925 15:24:19.670897 31163 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:19.670992 31163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:19.671265 31163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:19.674090 31163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "e3033e72644e4d348ed6f136aef7e03e"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-8vw7"
server_key: "413c2036dc26ba8be31a57fc3ec404b3"
server_key_iv: "2712e49399a54d3d493812e1b9f7066d"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.674779 31163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "e3033e72644e4d348ed6f136aef7e03e"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-8vw7"
server_key: "413c2036dc26ba8be31a57fc3ec404b3"
server_key_iv: "2712e49399a54d3d493812e1b9f7066d"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.679557 31163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:24:19.682796 31178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.684370 31163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:19.684572 31163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "e3033e72644e4d348ed6f136aef7e03e"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-8vw7"
server_key: "413c2036dc26ba8be31a57fc3ec404b3"
server_key_iv: "2712e49399a54d3d493812e1b9f7066d"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.684794 31163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:19.701319 31163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:19.705260 31163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:19.705534 31163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:19.723043 31163 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:40943
I20260925 15:24:19.723063 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:40943 every 8 connection(s)
I20260925 15:24:19.724398 31163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:19.728344 31230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:19.734424 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31163
I20260925 15:24:19.734608 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:19.735061 9097 external_mini_cluster.cc:1428] Setting key 6b160a1cf60c90a1c9307dd614ee2e99
I20260925 15:24:19.739861 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e: Bootstrap starting.
I20260925 15:24:19.743691 31230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:19.745050 31230 log.cc:824] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:19.748257 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e: No bootstrap required, opened a new log
Sep 25 15:24:19 dist-test-slave-8vw7 krb5kdc[31150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:24:19.758514 31230 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3033e72644e4d348ed6f136aef7e03e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 40943 } }
I20260925 15:24:19.759018 31230 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:19.759140 31230 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3033e72644e4d348ed6f136aef7e03e, State: Initialized, Role: FOLLOWER
I20260925 15:24:19.759900 31230 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3033e72644e4d348ed6f136aef7e03e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 40943 } }
I20260925 15:24:19.760195 31230 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:19.760334 31230 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:19.760510 31230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:19.761629 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.737076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45809 (local address 127.8.226.126:40943)
0925 15:24:19.737920 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.737937 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.737984 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:24:19.740307 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.740334 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.740853 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.741338 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.742627 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.744795 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.746692 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.747719 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.747895 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.752206 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.752216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.752235 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.752285 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.756424 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.757343 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.757357 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.757366 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.757571 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.757988 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.757993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.757996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.758738 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.759067 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.759706 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.760158 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":354,"threads_started":1}
I20260925 15:24:19.763756 31230 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3033e72644e4d348ed6f136aef7e03e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 40943 } }
I20260925 15:24:19.764344 31230 leader_election.cc:302] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3033e72644e4d348ed6f136aef7e03e; no voters:
I20260925 15:24:19.765061 31230 leader_election.cc:288] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:19.765182 31235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:19.765786 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:19.765867 31235 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [term 1 LEADER]: Becoming Leader. State: Replica: e3033e72644e4d348ed6f136aef7e03e, State: Running, Role: LEADER
I20260925 15:24:19.766428 31235 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3033e72644e4d348ed6f136aef7e03e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 40943 } }
I20260925 15:24:19.769207 31237 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3033e72644e4d348ed6f136aef7e03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3033e72644e4d348ed6f136aef7e03e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 40943 } } }
I20260925 15:24:19.769636 31237 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:19.769829 31235 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3033e72644e4d348ed6f136aef7e03e. Latest consensus state: current_term: 1 leader_uuid: "e3033e72644e4d348ed6f136aef7e03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3033e72644e4d348ed6f136aef7e03e" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 40943 } } }
I20260925 15:24:19.770150 31235 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:19.770550 31230 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:19.781260 31240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:19.787470 31240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:19.802871 31240 catalog_manager.cc:1347] Generated new cluster ID: 095533be01f747eaa7f7ab8dd1422640
I20260925 15:24:19.802992 31240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:19.815970 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.801123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46115 (local address 127.8.226.126:40943)
0925 15:24:19.801285 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.801291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.803547 (+ 2256us) server_negotiation.cc:407] Connection header received
0925 15:24:19.803761 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.803769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.803844 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.803949 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.805377 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.806139 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.807511 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.808035 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.808186 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.810431 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.810437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.810441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.810477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.813295 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.813951 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.813956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.813960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.814123 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.814490 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.814499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.814501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.814840 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.815099 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.815391 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.815597 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260925 15:24:19.822052 31240 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:19.823534 31240 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:19.836076 31240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3033e72644e4d348ed6f136aef7e03e: Generated new TSK 0
I20260925 15:24:19.837077 31240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:19.845768 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31163
I20260925 15:24:19.846701 31225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:19.936237 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31163
2026-09-25T15:24:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:20 dist-test-slave-8vw7 krb5kdc[31262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:20 dist-test-slave-8vw7 krb5kdc[31262](info): set up 2 sockets
Sep 25 15:24:20 dist-test-slave-8vw7 krb5kdc[31262](info): commencing operation
krb5kdc: starting...
W20260925 15:24:21.038753 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:21 dist-test-slave-8vw7 krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:21.280296 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39193
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:21.595132 31275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:21.595425 31275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:21.595515 31275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:21.602336 31275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:21.602455 31275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:21.602500 31275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:21.602552 31275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:21.602589 31275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:21.611960 31275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:39193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:21.613137 31275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:21.614409 31275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:21.625988 31283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:21.625944 31281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:21.626027 31280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:21.630396 31275 server_base.cc:1034] running on GCE node
I20260925 15:24:21.631176 31275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:21.632524 31275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:21.633720 31275 hybrid_clock.cc:648] HybridClock initialized: now 1790349861633686 us; error 30 us; skew 500 ppm
I20260925 15:24:21.634122 31275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:21 dist-test-slave-8vw7 krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349861, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:21.637275 31275 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:21.638760 31275 webserver.cc:460] Webserver started at http://127.8.226.126:38179/ using document root <none> and password file <none>
I20260925 15:24:21.639451 31275 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:21.639593 31275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:21.639894 31275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:21.642856 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "4242838312724aa3ac27ec47160b5ce5"
format_stamp: "Formatted at 2026-09-25 15:24:21 on dist-test-slave-8vw7"
server_key: "c50d61fe9b351275c20b7ceb74c72487"
server_key_iv: "4a774fae46b6daaa664c215ec882345d"
server_key_version: "encryptionkey@0"
I20260925 15:24:21.643610 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "4242838312724aa3ac27ec47160b5ce5"
format_stamp: "Formatted at 2026-09-25 15:24:21 on dist-test-slave-8vw7"
server_key: "c50d61fe9b351275c20b7ceb74c72487"
server_key_iv: "4a774fae46b6daaa664c215ec882345d"
server_key_version: "encryptionkey@0"
I20260925 15:24:21.648510 31275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:24:21.651809 31290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:21.653249 31275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:21.653460 31275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "4242838312724aa3ac27ec47160b5ce5"
format_stamp: "Formatted at 2026-09-25 15:24:21 on dist-test-slave-8vw7"
server_key: "c50d61fe9b351275c20b7ceb74c72487"
server_key_iv: "4a774fae46b6daaa664c215ec882345d"
server_key_version: "encryptionkey@0"
I20260925 15:24:21.653707 31275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:21.715615 31275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:21.720106 31275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:21.720345 31275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:21.739192 31275 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:39193
I20260925 15:24:21.739226 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:39193 every 8 connection(s)
I20260925 15:24:21.740588 31275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:21.743211 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31275
I20260925 15:24:21.743404 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:21.743973 9097 external_mini_cluster.cc:1428] Setting key ef274bd4b11f385fe82156c15eed0ead
I20260925 15:24:21.745266 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:21.757464 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:21.745820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47089 (local address 127.8.226.126:39193)
0925 15:24:21.746602 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:21.746614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:21.746653 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:24:21.748264 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:21.748291 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:21.748686 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:21.749098 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:21.749640 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:21.750987 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:21.752888 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:21.753634 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:21.753764 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:21.755031 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:21.755038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:21.755052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:21.755621 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:21.755694 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:21.755721 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:21.755817 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:21.756053 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:24:21.756369 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260925 15:24:21.759195 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5: Bootstrap starting.
I20260925 15:24:21.761525 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:21.762738 31342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:21.763743 31342 log.cc:824] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:21.767095 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5: No bootstrap required, opened a new log
I20260925 15:24:21.773070 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:21.765886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33419 (local address 127.8.226.126:39193)
0925 15:24:21.766078 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:21.766085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:21.766693 (+ 608us) server_negotiation.cc:407] Connection header received
0925 15:24:21.766943 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:21.766947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:21.767008 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:21.767097 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:21.767745 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:21.768490 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:21.770506 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:21.771010 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:21.771109 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:21.772279 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:21.772285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:21.772292 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:21.772309 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:21.772347 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:21.772365 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:21.772438 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:21.772585 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:24:21.772655 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:21.776782 31342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4242838312724aa3ac27ec47160b5ce5" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39193 } }
I20260925 15:24:21.777251 31342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:21.777350 31342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4242838312724aa3ac27ec47160b5ce5, State: Initialized, Role: FOLLOWER
I20260925 15:24:21.777974 31342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4242838312724aa3ac27ec47160b5ce5" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39193 } }
I20260925 15:24:21.778185 31342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:21.778352 31342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:21.778559 31342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:21.780836 31342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4242838312724aa3ac27ec47160b5ce5" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39193 } }
I20260925 15:24:21.781440 31342 leader_election.cc:302] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4242838312724aa3ac27ec47160b5ce5; no voters:
I20260925 15:24:21.782043 31342 leader_election.cc:288] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:21.782199 31352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:21.782850 31352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 4242838312724aa3ac27ec47160b5ce5, State: Running, Role: LEADER
I20260925 15:24:21.783406 31352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4242838312724aa3ac27ec47160b5ce5" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39193 } }
I20260925 15:24:21.784399 31342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:21.786463 31353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4242838312724aa3ac27ec47160b5ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4242838312724aa3ac27ec47160b5ce5" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39193 } } }
I20260925 15:24:21.786438 31354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4242838312724aa3ac27ec47160b5ce5. Latest consensus state: current_term: 1 leader_uuid: "4242838312724aa3ac27ec47160b5ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4242838312724aa3ac27ec47160b5ce5" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 39193 } } }
I20260925 15:24:21.786876 31354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:21.786876 31353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:21.787767 31358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:21.792557 31358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:21.802901 31358 catalog_manager.cc:1347] Generated new cluster ID: 50e3e724af194911b89d6f5188502f3b
I20260925 15:24:21.803020 31358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:21.815328 31358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:21.816787 31358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:21.835999 31358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4242838312724aa3ac27ec47160b5ce5: Generated new TSK 0
I20260925 15:24:21.837042 31358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:21.851799 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31275
W20260925 15:24:21.853637 31337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47089
W20260925 15:24:21.854365 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31275: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:21.854519 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31275
2026-09-25T15:24:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:21 dist-test-slave-8vw7 krb5kdc[31373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:21 dist-test-slave-8vw7 krb5kdc[31373](info): set up 2 sockets
Sep 25 15:24:21 dist-test-slave-8vw7 krb5kdc[31373](info): commencing operation
krb5kdc: starting...
W20260925 15:24:22.965646 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:23 dist-test-slave-8vw7 krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:23.234014 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:46809
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:23.560429 31386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:23.560711 31386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:23.560871 31386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:23.568001 31386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:23.568107 31386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:23.568249 31386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:23.568341 31386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:23.568408 31386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:23.578836 31386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:23.580103 31386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:23.581465 31386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:23.593076 31391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.593091 31394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.593305 31392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:23.594404 31386 server_base.cc:1034] running on GCE node
I20260925 15:24:23.595222 31386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:23.596576 31386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:23.597784 31386 hybrid_clock.cc:648] HybridClock initialized: now 1790349863597748 us; error 36 us; skew 500 ppm
I20260925 15:24:23.598137 31386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:23 dist-test-slave-8vw7 krb5kdc[31373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349863, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:23.606217 31386 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:23.607725 31386 webserver.cc:460] Webserver started at http://127.8.226.126:33785/ using document root <none> and password file <none>
I20260925 15:24:23.608420 31386 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:23.608570 31386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:23.608871 31386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:23.611876 31386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "3e826418c8b4460194c6c30dc62f85fb"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-8vw7"
server_key: "f68d04c304ab5ed67577289aa252e91b"
server_key_iv: "7d32bd116d54631f24b74b7818046190"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.612663 31386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "3e826418c8b4460194c6c30dc62f85fb"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-8vw7"
server_key: "f68d04c304ab5ed67577289aa252e91b"
server_key_iv: "7d32bd116d54631f24b74b7818046190"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.618013 31386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:24:23.621464 31401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.622797 31386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:24:23.622982 31386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "3e826418c8b4460194c6c30dc62f85fb"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-8vw7"
server_key: "f68d04c304ab5ed67577289aa252e91b"
server_key_iv: "7d32bd116d54631f24b74b7818046190"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.623193 31386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:23.628803 31386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:23.632608 31386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:23.632894 31386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:23.650916 31386 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:46809
I20260925 15:24:23.650949 31452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:46809 every 8 connection(s)
I20260925 15:24:23.652278 31386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:23.655892 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:23.658555 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31386
I20260925 15:24:23.658751 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:23.659170 9097 external_mini_cluster.cc:1428] Setting key dca72ee92e8174fc5f5d02b08878c331
I20260925 15:24:23.668447 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb: Bootstrap starting.
I20260925 15:24:23.668552 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.660977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52017 (local address 127.8.226.126:46809)
0925 15:24:23.662020 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.662037 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.662087 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:24:23.664773 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.664796 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.665253 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.665675 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.666817 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.666824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.666837 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.666851 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.666923 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.666946 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.667203 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.667348 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.667542 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":421,"threads_started":1}
I20260925 15:24:23.672209 31453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:23.672762 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:23.673441 31453 log.cc:824] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:23.676522 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb: No bootstrap required, opened a new log
I20260925 15:24:23.680171 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.677593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37461 (local address 127.8.226.126:46809)
0925 15:24:23.677769 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.677777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.678152 (+ 375us) server_negotiation.cc:407] Connection header received
0925 15:24:23.678365 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.678369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.678416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.678538 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.679495 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.679508 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.679511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.679526 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.679566 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.679581 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.679713 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.679845 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.679909 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:24:23.685660 31453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e826418c8b4460194c6c30dc62f85fb" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46809 } }
I20260925 15:24:23.686111 31453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:23.686235 31453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e826418c8b4460194c6c30dc62f85fb, State: Initialized, Role: FOLLOWER
I20260925 15:24:23.686829 31453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e826418c8b4460194c6c30dc62f85fb" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46809 } }
I20260925 15:24:23.687093 31453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:23.687229 31453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:23.687434 31453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:23.690085 31453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e826418c8b4460194c6c30dc62f85fb" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46809 } }
I20260925 15:24:23.690708 31453 leader_election.cc:302] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e826418c8b4460194c6c30dc62f85fb; no voters:
I20260925 15:24:23.691419 31453 leader_election.cc:288] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:23.691534 31463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:23.692194 31463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [term 1 LEADER]: Becoming Leader. State: Replica: 3e826418c8b4460194c6c30dc62f85fb, State: Running, Role: LEADER
I20260925 15:24:23.692821 31463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e826418c8b4460194c6c30dc62f85fb" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46809 } }
I20260925 15:24:23.693842 31453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:23.695870 31464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e826418c8b4460194c6c30dc62f85fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e826418c8b4460194c6c30dc62f85fb" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46809 } } }
I20260925 15:24:23.696266 31464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:23.698447 31469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:23.701164 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e826418c8b4460194c6c30dc62f85fb. Latest consensus state: current_term: 1 leader_uuid: "3e826418c8b4460194c6c30dc62f85fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e826418c8b4460194c6c30dc62f85fb" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 46809 } } }
I20260925 15:24:23.701571 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:23.703188 31469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:23.712184 31469 catalog_manager.cc:1347] Generated new cluster ID: b60e0352169b4ef7a87076e66af8f328
I20260925 15:24:23.712311 31469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:23.724588 31469 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:23.725097 31469 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:23.732151 31469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e826418c8b4460194c6c30dc62f85fb: Generated new TSK 0
I20260925 15:24:23.733209 31469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:23.740808 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31386
W20260925 15:24:23.742087 31448 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52017
W20260925 15:24:23.742744 9097 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31386: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:24:23.742893 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31386
2026-09-25T15:24:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1887 ms)
[----------] 18 tests from AuthTokenIssuingTest (33748 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:23 dist-test-slave-8vw7 krb5kdc[31484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:23 dist-test-slave-8vw7 krb5kdc[31484](info): set up 2 sockets
Sep 25 15:24:23 dist-test-slave-8vw7 krb5kdc[31484](info): commencing operation
krb5kdc: starting...
W20260925 15:24:24.873672 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:25 dist-test-slave-8vw7 krb5kdc[31484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:25.130162 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:45089
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:45089,127.8.226.125:34903,127.8.226.124:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:25.453343 31497 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:25.453706 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:25.453811 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:25.453963 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:25.460803 31497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:25.460871 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:25.460924 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:25.461011 31497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:25.461119 31497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:25.470929 31497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:45089,127.8.226.125:34903,127.8.226.124:36489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:25.472306 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:25.473630 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:25.485010 31503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:25.485018 31505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:25.485026 31502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:25.486006 31497 server_base.cc:1034] running on GCE node
I20260925 15:24:25.486716 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:25.488013 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:25.489244 31497 hybrid_clock.cc:648] HybridClock initialized: now 1790349865489172 us; error 48 us; skew 500 ppm
I20260925 15:24:25.489545 31497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:25 dist-test-slave-8vw7 krb5kdc[31484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:25.492580 31497 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:25.493942 31497 webserver.cc:460] Webserver started at http://127.8.226.126:39455/ using document root <none> and password file <none>
I20260925 15:24:25.494596 31497 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:25.494721 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:25.495035 31497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:25.497932 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "7db30113265d4a88a8fe133553d3c8dd"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-8vw7"
server_key: "1f8347e090cf4839adaf13f96d345a10"
server_key_iv: "b4dbaf44a40cc3a8296fe84ef29b3b76"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.498665 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "7db30113265d4a88a8fe133553d3c8dd"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-8vw7"
server_key: "1f8347e090cf4839adaf13f96d345a10"
server_key_iv: "b4dbaf44a40cc3a8296fe84ef29b3b76"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.503453 31497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:24:25.506726 31512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:25.507943 31497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:24:25.508117 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "7db30113265d4a88a8fe133553d3c8dd"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-8vw7"
server_key: "1f8347e090cf4839adaf13f96d345a10"
server_key_iv: "b4dbaf44a40cc3a8296fe84ef29b3b76"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.508337 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:25.525321 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:25.528925 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:25.529250 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:25.546761 31497 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:45089
I20260925 15:24:25.546792 31563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:45089 every 8 connection(s)
I20260925 15:24:25.548036 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:25.551513 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:25.553712 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31497
I20260925 15:24:25.553941 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:25.554751 9097 external_mini_cluster.cc:1428] Setting key 35a96dcabae56213878539d3471e703a
I20260925 15:24:25.560545 31564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:25 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.125@KRBTEST.COM" created.
I20260925 15:24:25.584682 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.564238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:45089 (local address 127.0.0.1:54438)
0925 15:24:25.565153 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.565203 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:25.567233 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:25.568102 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:25.568106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:25.568628 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:25.568944 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:25.569015 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.571244 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.571248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:25.572275 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:25.572286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.573579 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.573584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:25.573704 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:25.574355 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:25.574375 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:25.576402 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:25.580684 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:25.580693 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:25.580701 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:25.581140 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:25.581612 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:25.581616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:25.581618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:25.581775 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:25.582647 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:25.582653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:25.582884 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 15:24:25.583155 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 15:24:25.583523 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":373,"threads_started":1}
I20260925 15:24:25.584690 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.564238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.8.226.126:45089)
0925 15:24:25.566047 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:25.566052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:25.566121 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:24:25.567585 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:25.567595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:25.567876 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:25.568078 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:25.569213 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.571088 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.572412 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.573382 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.573562 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:25.576627 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:25.576639 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:25.576656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:25.576700 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:25.580555 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:25.581273 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:25.581282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:25.581285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:25.581483 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:25.581896 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:25.581904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:25.581909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:25.582519 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:25.582675 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:25.583297 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 15:24:25.583650 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":1625,"spinlock_wait_cycles":47872,"thread_start_us":1423,"threads_started":1}
I20260925 15:24:25.587725 31564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:25.588650 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.588289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46582)
0925 15:24:25.588447 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.588545 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:24:25.588873 31514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
I20260925 15:24:25.590425 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.590093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46592)
0925 15:24:25.590240 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.590300 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:24:25.590608 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:34903: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab.
Entry for principal kudu/127.8.226.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab.
I20260925 15:24:25.613232 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:34903
--webserver_interface=127.8.226.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab
--principal=kudu/127.8.226.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:45089,127.8.226.125:34903,127.8.226.124:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:25.639935 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } attempt: 1
I20260925 15:24:25.641209 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.640641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46594)
0925 15:24:25.640878 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.641030 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 15:24:25.642194 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.641823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46598)
0925 15:24:25.641958 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.642035 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:24:25.642405 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:34903: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
I20260925 15:24:25.710690 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } attempt: 2
I20260925 15:24:25.711930 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.711421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46608)
0925 15:24:25.711660 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.711761 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 15:24:25.713007 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.712604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46618)
0925 15:24:25.712755 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.712841 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 15:24:25.713189 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:34903: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
I20260925 15:24:25.804517 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } attempt: 3
I20260925 15:24:25.805652 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.805236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46626)
0925 15:24:25.805432 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.805526 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:24:25.806651 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.806309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46640)
0925 15:24:25.806464 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.806527 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:24:25.806842 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:34903: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
W20260925 15:24:25.935935 31571 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:25.936297 31571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:25.936424 31571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:25.936563 31571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:25.944083 31571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:25.944211 31571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:25.944345 31571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:25.944437 31571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:25.944550 31571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:25.950054 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } attempt: 4
I20260925 15:24:25.951228 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.950718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46642)
0925 15:24:25.950991 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.951088 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 15:24:25.952219 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.951876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46654)
0925 15:24:25.952015 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:25.952099 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:24:25.952399 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:34903: Network error: Client connection negotiation failed: client connection to 127.8.226.125:34903: connect: Connection refused (error 111)
I20260925 15:24:25.955081 31571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:45089,127.8.226.125:34903,127.8.226.124:36489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab
--principal=kudu/127.8.226.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.226.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:25.956346 31571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:25.957741 31571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:25.969544 31576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:25.969472 31577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:25.969507 31579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:25.970248 31571 server_base.cc:1034] running on GCE node
I20260925 15:24:25.971007 31571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:25.972380 31571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:25.973634 31571 hybrid_clock.cc:648] HybridClock initialized: now 1790349865973553 us; error 49 us; skew 500 ppm
I20260925 15:24:25.973973 31571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:25 dist-test-slave-8vw7 krb5kdc[31484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:25.977106 31571 init.cc:377] Logged in from keytab as kudu/127.8.226.125@KRBTEST.COM (short username kudu)
I20260925 15:24:25.978483 31571 webserver.cc:460] Webserver started at http://127.8.226.125:34107/ using document root <none> and password file <none>
I20260925 15:24:25.979167 31571 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:25.979296 31571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:25.979610 31571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:25.982558 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data/instance:
uuid: "dcad6574e0fd418893f2e14d79d0d341"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-8vw7"
server_key: "a48d89bf8137521d00a3da69a6c14ad8"
server_key_iv: "c570a119e05fbf421c78fee8658c4a7c"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.983340 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal/instance:
uuid: "dcad6574e0fd418893f2e14d79d0d341"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-8vw7"
server_key: "a48d89bf8137521d00a3da69a6c14ad8"
server_key_iv: "c570a119e05fbf421c78fee8658c4a7c"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.988372 31571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:24:25.991681 31586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:25.992869 31571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:25.993072 31571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal
uuid: "dcad6574e0fd418893f2e14d79d0d341"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-8vw7"
server_key: "a48d89bf8137521d00a3da69a6c14ad8"
server_key_iv: "c570a119e05fbf421c78fee8658c4a7c"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.993299 31571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:26.029032 31571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:26.033067 31571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:26.033349 31571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:26.051685 31571 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.125:34903
I20260925 15:24:26.051692 31637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.125:34903 every 8 connection(s)
I20260925 15:24:26.053022 31571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
I20260925 15:24:26.056586 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:26.056883 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31571
I20260925 15:24:26.057085 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-1/wal/instance
I20260925 15:24:26.057515 9097 external_mini_cluster.cc:1428] Setting key 8ea7a395ab1d78372a89f0438ceb60f2
I20260925 15:24:26.062929 31638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.124@KRBTEST.COM" created.
I20260925 15:24:26.083928 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.065558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.8.226.126:45089)
0925 15:24:26.065776 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.065783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.066918 (+ 1135us) server_negotiation.cc:407] Connection header received
0925 15:24:26.068715 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.068724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.068813 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.068936 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.070396 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.071507 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.072933 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.073706 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.073846 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.077209 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.077218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.077226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.077274 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.081065 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.081721 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.081725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.081727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.081877 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.082289 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.082291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.082293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.082556 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.082675 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.083450 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.083708 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:24:26.085248 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.065981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:45089 (local address 127.0.0.1:54454)
0925 15:24:26.066727 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.066782 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.068454 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.069166 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.069173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.069682 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.070214 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.070231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.071648 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.071652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.072792 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.072802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.073896 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.073900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.074088 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.074784 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.074808 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.076915 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.081190 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.081196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.081207 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.081610 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.082000 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.082007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.082014 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.082192 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.082659 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.082665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.083049 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.083327 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.083804 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":321,"threads_started":1}
I20260925 15:24:26.087006 31638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:24:26.103147 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.087927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.8.226.125:34903)
0925 15:24:26.088633 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.088643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.088676 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:24:26.088738 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.088748 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.089002 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.089140 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.089831 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.091533 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.092656 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.093257 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.093351 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.097267 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.097276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.097288 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.097320 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.100105 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.100870 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.100876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.100878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.101037 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.101488 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.101492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.101495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.101896 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.102065 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.102494 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.102826 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":514,"thread_start_us":366,"threads_started":1}
I20260925 15:24:26.103175 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.087644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46668)
0925 15:24:26.087849 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.087869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.087972 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.089156 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.089165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.089506 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.089657 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.089666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.091672 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.091675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.092517 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.092527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.093405 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.093410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.093631 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.094223 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.094250 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.097040 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.100231 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.100238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.100245 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.100758 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.101138 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.101145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.101152 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.101355 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.102058 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.102065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.102230 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.102386 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.102709 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:24:26.105858 31638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:26.107112 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.106506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59826)
0925 15:24:26.106744 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.106898 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 15:24:26.107370 31588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.108922 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.108530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59830)
0925 15:24:26.108697 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.108794 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:24:26.109144 31638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab.
Entry for principal kudu/127.8.226.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab.
I20260925 15:24:26.117782 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:36489
--webserver_interface=127.8.226.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab
--principal=kudu/127.8.226.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:45089,127.8.226.125:34903,127.8.226.124:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:26.158491 31638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 1
I20260925 15:24:26.159816 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.159228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59834)
0925 15:24:26.159455 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.159597 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:24:26.160872 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.160462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59844)
0925 15:24:26.160621 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.160717 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:24:26.161084 31638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.229321 31638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 2
I20260925 15:24:26.230556 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.230041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59860)
0925 15:24:26.230263 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.230372 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:24:26.231518 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.231171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59868)
0925 15:24:26.231308 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.231387 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:24:26.231729 31638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.251709 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } attempt: 5
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:24:26.264390 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.252614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.8.226.125:34903)
0925 15:24:26.252810 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.252816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.252850 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:24:26.252934 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.252938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.253057 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.253161 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.253916 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.254605 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.255743 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.256200 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.256328 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.259371 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.259377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.259387 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.259417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.261964 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.262581 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.262588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.262591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.262722 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.263076 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.263082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.263085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.263348 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.263496 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.263875 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.264079 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:24:26.264392 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.252429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46672)
0925 15:24:26.252613 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.252641 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.252790 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.253199 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.253220 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.253602 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.253767 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.253775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.254736 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.254744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.255594 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.255613 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.256349 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.256354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.256571 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.257265 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.257282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.259192 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.262092 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.262098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.262103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.262466 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.262831 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.262835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.262839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.262971 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.263473 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.263478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.263635 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.263776 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.264041 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:24:26.265736 31564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:26.266662 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.266287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59880)
0925 15:24:26.266433 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.266526 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:24:26.267549 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.267258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59894)
0925 15:24:26.267392 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.267462 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:24:26.267769 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.318969 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 1
I20260925 15:24:26.320258 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.319604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59896)
0925 15:24:26.319877 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.320018 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:24:26.321458 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.321010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59906)
0925 15:24:26.321185 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.321292 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:24:26.321712 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.322950 31638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 3
I20260925 15:24:26.323942 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.323476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59920)
0925 15:24:26.323648 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.323777 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:24:26.325033 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.324542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59922)
0925 15:24:26.324773 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.324868 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260925 15:24:26.325230 31638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.389958 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 2
I20260925 15:24:26.391245 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.390646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59938)
0925 15:24:26.390882 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.391000 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:24:26.392318 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.391917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59952)
0925 15:24:26.392079 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.392178 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:24:26.392517 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
W20260925 15:24:26.443403 31645 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:26.443742 31645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:26.443809 31645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:26.443918 31645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:26.450834 31645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:26.450917 31645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:26.450973 31645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:26.451023 31645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:26.451077 31645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:26.460354 31645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:45089,127.8.226.125:34903,127.8.226.124:36489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab
--principal=kudu/127.8.226.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.226.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:26.461536 31645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:26.462743 31645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:24:26.468431 31638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 4
I20260925 15:24:26.469631 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.469178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59960)
0925 15:24:26.469390 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.469473 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 15:24:26.470615 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.470247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59972)
0925 15:24:26.470410 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.470490 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 15:24:26.470814 31638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
W20260925 15:24:26.474682 31651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:26.474794 31650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:26.474838 31653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:26.475778 31645 server_base.cc:1034] running on GCE node
I20260925 15:24:26.476478 31645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:26.477872 31645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:26.479110 31645 hybrid_clock.cc:648] HybridClock initialized: now 1790349866479034 us; error 50 us; skew 500 ppm
I20260925 15:24:26.479434 31645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:26.482633 31645 init.cc:377] Logged in from keytab as kudu/127.8.226.124@KRBTEST.COM (short username kudu)
I20260925 15:24:26.484151 31645 webserver.cc:460] Webserver started at http://127.8.226.124:45809/ using document root <none> and password file <none>
I20260925 15:24:26.484849 31645 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:26.485006 31645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:26.485390 31645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:26.488399 31645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data/instance:
uuid: "735d4abf83304bfb9f45b8e849da9722"
format_stamp: "Formatted at 2026-09-25 15:24:26 on dist-test-slave-8vw7"
server_key: "c39511cae2d758821415566111851a92"
server_key_iv: "fd33f62d33e09faf28ccd184af5792a8"
server_key_version: "encryptionkey@0"
I20260925 15:24:26.489307 31645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal/instance:
uuid: "735d4abf83304bfb9f45b8e849da9722"
format_stamp: "Formatted at 2026-09-25 15:24:26 on dist-test-slave-8vw7"
server_key: "c39511cae2d758821415566111851a92"
server_key_iv: "fd33f62d33e09faf28ccd184af5792a8"
server_key_version: "encryptionkey@0"
I20260925 15:24:26.494465 31645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:24:26.497978 31660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:26.498750 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 3
I20260925 15:24:26.499511 31645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:24:26.499696 31645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal
uuid: "735d4abf83304bfb9f45b8e849da9722"
format_stamp: "Formatted at 2026-09-25 15:24:26 on dist-test-slave-8vw7"
server_key: "c39511cae2d758821415566111851a92"
server_key_iv: "fd33f62d33e09faf28ccd184af5792a8"
server_key_version: "encryptionkey@0"
I20260925 15:24:26.499897 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.499328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59974)
0925 15:24:26.499595 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.499707 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:24:26.499931 31645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:26.500943 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.500538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:59988)
0925 15:24:26.500710 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.500804 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:24:26.501199 31564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:36489: Network error: Client connection negotiation failed: client connection to 127.8.226.124:36489: connect: Connection refused (error 111)
I20260925 15:24:26.525493 31645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:26.529780 31645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:26.530055 31645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:26.548202 31645 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.124:36489
I20260925 15:24:26.548231 31711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.124:36489 every 8 connection(s)
I20260925 15:24:26.549713 31645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
I20260925 15:24:26.551862 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31645
I20260925 15:24:26.552053 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349523192742-9097-0/minicluster-data/master-2/wal/instance
I20260925 15:24:26.552573 9097 external_mini_cluster.cc:1428] Setting key e9bf3be0c8fd72a83e3f7c4b3baf30b8
I20260925 15:24:26.553999 31712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:26.563748 31712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:24:26.580435 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.554366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.8.226.124:36489)
0925 15:24:26.559703 (+ 5337us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.559714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.559752 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:24:26.561527 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.561546 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.561917 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.562316 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.563698 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.565300 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.567136 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.567907 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.568031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.573141 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.573148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.573158 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.573189 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.576228 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.576982 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.576991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.576999 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.577169 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.577554 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.577557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.577576 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.578219 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.578501 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.578865 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.579274 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":5153,"thread_start_us":4972,"threads_started":1}
I20260925 15:24:26.581591 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.566178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:45089 (local address 127.0.0.1:54470)
0925 15:24:26.566789 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.566820 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.566991 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.567377 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.567382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.568053 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.568431 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.568446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.569635 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.569641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.571301 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.571313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.572255 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.572265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.572463 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.573096 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.573121 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.575519 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.579253 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.579261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.579266 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.579648 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.580032 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.580034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.580037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.580166 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.580766 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.580772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.580901 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.581132 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.581325 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":25,"thread_start_us":311,"threads_started":1}
I20260925 15:24:26.581839 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.566115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.8.226.126:45089)
0925 15:24:26.566283 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.566287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.566918 (+ 631us) server_negotiation.cc:407] Connection header received
0925 15:24:26.567121 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.567126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.567203 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.567302 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.568623 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.569468 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.571425 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.572047 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.572137 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.575716 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.575726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.575731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.575770 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.579118 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.579765 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.579770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.579773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.579927 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.580266 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.580271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.580274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.580599 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.580773 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.581230 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.581431 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:24:26.583063 31712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:24:26.596434 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.583749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.8.226.125:34903)
0925 15:24:26.583913 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.583920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.583937 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:24:26.584053 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.584058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.584136 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.584236 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.585085 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.585870 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.587306 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.587744 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.587834 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.590977 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.590986 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.590994 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.591035 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.593913 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.594640 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.594643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.594645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.594781 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.595173 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.595176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.595178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.595413 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.595517 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.595984 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.596174 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:24:26.596495 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.583634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:34903 (local address 127.0.0.1:46678)
0925 15:24:26.583853 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.583868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.583949 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.584281 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.584287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.584742 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.584932 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.584946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.586011 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.586016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.587159 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.587175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.587882 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.587890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.588069 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.588600 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.588616 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.590796 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.594048 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.594055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.594059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.594535 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.594886 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.594892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.594898 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.595066 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.595525 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.595531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.595694 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.595857 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.596156 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 15:24:26.597648 31712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:26.598240 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.583566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.8.226.126:45089)
0925 15:24:26.583913 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.583919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.583937 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:24:26.583984 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.583988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.584054 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.584149 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.585592 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.586430 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.587963 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.588339 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.588468 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.592551 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.592556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.592559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.592585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.595424 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.596237 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.596242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.596246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.596374 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.596808 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.596813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.596817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.597197 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.597472 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.597739 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.597937 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349866, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:24:26.610733 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.598313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:60004)
0925 15:24:26.598562 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.598579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.598667 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.598979 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.598982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.599357 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.599497 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.599511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.600863 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.600867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.601909 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.601922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.602541 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.602548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.602727 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.603267 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.603288 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.605670 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.608456 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.608464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.608468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.608801 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.609178 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.609182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.609187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.609311 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.609774 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.609779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.609911 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.610062 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.610343 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 15:24:26.610808 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.598516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.8.226.124:36489)
0925 15:24:26.598675 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.598680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.598734 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:24:26.598781 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.598788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.598864 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.598960 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.599664 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.600734 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.602036 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.602404 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.602496 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.605856 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.605860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.605863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.605889 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.608334 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.608909 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.608916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.608919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.609080 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.609407 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.609415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.609418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.609668 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.609807 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.610133 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.610486 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":52}
I20260925 15:24:26.613731 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.599906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.8.226.125:34903)
0925 15:24:26.600069 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.600072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.600086 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:24:26.600126 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.600129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.600182 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.600251 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.601596 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.602444 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.603907 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.604415 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.604537 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.608476 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.608480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.608483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.608511 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.611152 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.611831 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.611836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.611839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.612010 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.612463 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.612468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.612471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.612756 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.613010 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.613326 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.613475 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:24:26.616339 31712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: Bootstrap starting.
I20260925 15:24:26.618171 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:26.619596 31712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:26.620565 31712 log.cc:824] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:26.623250 31712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: No bootstrap required, opened a new log
I20260925 15:24:26.620641 31539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7db30113265d4a88a8fe133553d3c8dd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474
I20260925 15:24:26.630048 31712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.630367 31712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735d4abf83304bfb9f45b8e849da9722, State: Initialized, Role: FOLLOWER
I20260925 15:24:26.630797 31712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.631722 31718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.632104 31718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.632583 31712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:24:26.641263 31729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:26.641373 31729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:26.639467 31645 master_runner.cc:186] Error getting master registration for 127.8.226.126:45089: OK, instance_id { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" instance_seqno: 1790349865541785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:26.650471 31645 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:24:26.678421 31564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 4
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
I20260925 15:24:26.692263 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.679313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.8.226.124:36489)
0925 15:24:26.679541 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.679550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.679574 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:24:26.679676 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.679681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.679773 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.679891 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.680659 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.681577 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.682823 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.683280 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.683427 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.686788 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.686792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.686796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.686826 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.689660 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.690317 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.690323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.690326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.690441 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.690806 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.690813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.690816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.691158 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.691323 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.691689 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.691901 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:24:26.692257 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.679242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:60020)
0925 15:24:26.679430 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.679452 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.679579 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.679937 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.679943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.680348 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.680507 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.680516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.681719 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.681727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.682679 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.682690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.683442 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.683448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.683655 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.684334 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.684358 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.686602 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.689821 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.689827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.689831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.690193 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.690547 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.690552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.690556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.690689 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.691282 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.691287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.691452 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.691590 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.691866 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:24:26.697151 31564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd: Bootstrap starting.
I20260925 15:24:26.701061 31564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:26.702338 31564 log.cc:824] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:26.705777 31564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd: No bootstrap required, opened a new log
I20260925 15:24:26.713143 31564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.713534 31564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db30113265d4a88a8fe133553d3c8dd, State: Initialized, Role: FOLLOWER
I20260925 15:24:26.714010 31564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.715221 31732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.715775 31732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.716316 31564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:26.716368 31539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:24:26.716557 31539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:26.719991 31539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.721521 31539 leader_election.cc:288] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [CANDIDATE]: Term 1 election: Requested vote from peers dcad6574e0fd418893f2e14d79d0d341 (127.8.226.125:34903), 735d4abf83304bfb9f45b8e849da9722 (127.8.226.124:36489)
I20260925 15:24:26.723146 31687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7db30113265d4a88a8fe133553d3c8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "735d4abf83304bfb9f45b8e849da9722"
I20260925 15:24:26.723639 31687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:26.728091 31687 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db30113265d4a88a8fe133553d3c8dd in term 1.
I20260925 15:24:26.728828 31514 leader_election.cc:302] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 735d4abf83304bfb9f45b8e849da9722, 7db30113265d4a88a8fe133553d3c8dd; no voters:
I20260925 15:24:26.729393 31732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:26.730103 31732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [term 1 LEADER]: Becoming Leader. State: Replica: 7db30113265d4a88a8fe133553d3c8dd, State: Running, Role: LEADER
I20260925 15:24:26.730645 31732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.737342 31744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db30113265d4a88a8fe133553d3c8dd. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.738269 31744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:26.749684 31747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:26.757076 31497 master_runner.cc:186] Error getting master registration for 127.8.226.125:34903: OK, instance_id { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" instance_seqno: 1790349866046491 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:26.768168 31497 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:24:26.770031 31638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } attempt: 5
I20260925 15:24:26.774734 31747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:24:26 dist-test-slave-8vw7 krb5kdc[31484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
W20260925 15:24:26.781878 31518 tablet.cc:2369] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:26.782178 31666 tablet.cc:2369] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:26.790664 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.770531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:36489 (local address 127.0.0.1:60024)
0925 15:24:26.770891 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:26.770910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:26.771008 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:26.771537 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:26.771542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:26.772065 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:26.772253 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.772266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.773703 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.773709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.775132 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:26.775148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.776449 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.776456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:26.776702 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.777364 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:26.777388 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:26.780486 (+ 3098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:26.784611 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.784617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.784621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.785448 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.786173 (+ 725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:26.786178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:26.786182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:26.786491 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:26.787464 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:26.787469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:26.787731 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 15:24:26.788234 (+ 503us) client_negotiation.cc:240] Negotiation successful
0925 15:24:26.789231 (+ 997us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260925 15:24:26.791093 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.770918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.8.226.124:36489)
0925 15:24:26.771146 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.771155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.771176 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:24:26.771266 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.771274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.771384 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.771503 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.772417 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.773585 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.775371 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.776119 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.776236 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.780766 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.780773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.780778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.780821 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.784466 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.785624 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.785630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.785634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.785903 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.786638 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.786643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.786679 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.787286 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.787989 (+ 703us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.788559 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.789419 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:24:26.791865 31687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [term 1 FOLLOWER]: Refusing update from remote peer 7db30113265d4a88a8fe133553d3c8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:26.793068 31744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:26.799185 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: Bootstrap starting.
I20260925 15:24:26.803954 31638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:26.805440 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.752429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.8.226.126:45089)
0925 15:24:26.752680 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.752689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.761612 (+ 8923us) server_negotiation.cc:407] Connection header received
0925 15:24:26.762960 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.762966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.763052 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.763123 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.767254 (+ 4131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.768474 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.775824 (+ 7350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.776547 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.776638 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.791460 (+ 14822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.791467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.791471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.791509 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.795045 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.801732 (+ 6687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.801738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.801742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.801909 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.802535 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.802541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.802544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.802909 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.803209 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.804767 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.805092 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:24:26.807215 31638 log.cc:824] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:26.818811 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.762116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.8.226.125:34903)
0925 15:24:26.762327 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.762334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.771691 (+ 9357us) server_negotiation.cc:407] Connection header received
0925 15:24:26.773308 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.773313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.773378 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.773436 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.785611 (+ 12175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.786892 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.800001 (+ 13109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.800650 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.800770 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.810292 (+ 9522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.810299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.810303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.810340 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.814256 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.815493 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.815499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.815503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.815643 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.816288 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.816293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.816297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.816657 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.816897 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.818007 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.818306 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:26.822135 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: No bootstrap required, opened a new log
I20260925 15:24:26.823449 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.772203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.8.226.124:36489)
0925 15:24:26.773193 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.773199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.791889 (+ 18690us) server_negotiation.cc:407] Connection header received
0925 15:24:26.792048 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.792054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.792183 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.792413 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.795588 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.797950 (+ 2362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.801779 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.802866 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.803284 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.809760 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.809792 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.809798 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:26.809864 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.817041 (+ 7177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.818520 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.818550 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.818552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.819086 (+ 534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:26.819814 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:26.819818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:26.819856 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:26.820415 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.821633 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.821810 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.822097 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":467,"threads_started":1}
I20260925 15:24:26.826382 31718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db30113265d4a88a8fe133553d3c8dd. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.826679 31718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.829627 31756 mvcc.cc:205] Tried to move back new op lower bound from 7333273054366507008 to 7333273054138310656. Current Snapshot: MvccSnapshot[applied={T|T < 7333273054366507008}]
I20260925 15:24:26.832216 31757 mvcc.cc:205] Tried to move back new op lower bound from 7333273054366507008 to 7333273054138310656. Current Snapshot: MvccSnapshot[applied={T|T < 7333273054366507008}]
I20260925 15:24:26.844678 31718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.844892 31718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.845969 31638 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.846372 31638 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcad6574e0fd418893f2e14d79d0d341, State: Initialized, Role: FOLLOWER
I20260925 15:24:26.846880 31638 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } }
I20260925 15:24:26.848369 31762 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.849273 31762 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.854568 31638 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:26.854692 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7db30113265d4a88a8fe133553d3c8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dcad6574e0fd418893f2e14d79d0d341"
I20260925 15:24:26.855258 31613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:26.858566 31744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.858881 31744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:26.860299 31613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db30113265d4a88a8fe133553d3c8dd in term 1.
I20260925 15:24:26.860414 31747 catalog_manager.cc:1347] Generated new cluster ID: bb3767831105439eb636c0ba7573596c
I20260925 15:24:26.860510 31747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:26.861065 31612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [term 1 FOLLOWER]: Refusing update from remote peer 7db30113265d4a88a8fe133553d3c8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:26.863102 31732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:26.878611 31747 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:26.880941 31747 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:26.912575 31769 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.913219 31769 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.913101 31744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.913378 31744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:26.913717 31762 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db30113265d4a88a8fe133553d3c8dd. Latest consensus state: current_term: 1 leader_uuid: "7db30113265d4a88a8fe133553d3c8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db30113265d4a88a8fe133553d3c8dd" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 45089 } } peers { permanent_uuid: "dcad6574e0fd418893f2e14d79d0d341" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 34903 } } peers { permanent_uuid: "735d4abf83304bfb9f45b8e849da9722" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 36489 } } }
I20260925 15:24:26.914063 31762 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:26.947829 31747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db30113265d4a88a8fe133553d3c8dd: Generated new TSK 0
I20260925 15:24:26.949281 31747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:26.970631 31778 catalog_manager.cc:1259] Loaded cluster ID: bb3767831105439eb636c0ba7573596c
I20260925 15:24:26.970749 31778 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: loading cluster ID for follower catalog manager: success
I20260925 15:24:26.974663 31778 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: acquiring CA information for follower catalog manager: success
I20260925 15:24:26.977268 31778 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcad6574e0fd418893f2e14d79d0d341: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:24:27.086215 31567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.072203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.8.226.126:45089)
0925 15:24:27.072409 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.072416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.083118 (+ 10702us) server_negotiation.cc:407] Connection header received
0925 15:24:27.084527 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.084534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.084627 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.084717 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.085956 (+ 1239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54496: BlockingRecv error: recv got EOF from 127.0.0.1:54496 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 15:24:27.089804 31642 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.083676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.8.226.125:34903)
0925 15:24:27.083901 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.083907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.087785 (+ 3878us) server_negotiation.cc:407] Connection header received
0925 15:24:27.088487 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.088493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.088577 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.088651 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.089584 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46698: BlockingRecv error: recv got EOF from 127.0.0.1:46698 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:24:27.095088 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.092791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.8.226.124:36489)
0925 15:24:27.093011 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.093018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.093467 (+ 449us) server_negotiation.cc:407] Connection header received
0925 15:24:27.093701 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.093715 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.093804 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.093888 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.094870 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60042: BlockingRecv error: recv got EOF from 127.0.0.1:60042 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:24:27.147022 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.128058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.8.226.125:34903)
0925 15:24:27.128235 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.128241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.138901 (+ 10660us) server_negotiation.cc:407] Connection header received
0925 15:24:27.139480 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.139487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.139695 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.139847 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.140586 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.141798 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.143879 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.144586 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.144680 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.145462 (+ 782us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.146213 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.146341 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.146579 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.146670 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 15:24:27.194029 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.180150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.8.226.124:36489)
0925 15:24:27.180352 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.180359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.191451 (+ 11092us) server_negotiation.cc:407] Connection header received
0925 15:24:27.192014 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.192020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.192155 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.192243 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.193729 (+ 1486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60044: BlockingRecv error: recv got EOF from 127.0.0.1:60044 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:24:27.248859 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.231074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.8.226.125:34903)
0925 15:24:27.231272 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.231280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.241970 (+ 10690us) server_negotiation.cc:407] Connection header received
0925 15:24:27.242472 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.242479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.242597 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.242670 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.243369 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.244534 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.246568 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.247283 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.247381 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.248009 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.248160 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.248220 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.248443 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.248536 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 15:24:27.281481 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.279873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.8.226.124:36489)
0925 15:24:27.280045 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.280051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.280070 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:24:27.280147 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.280152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.280234 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.280352 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.281242 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60060: BlockingRecv error: recv got EOF from 127.0.0.1:60060 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:24:27.294860 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.287729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.8.226.125:34903)
0925 15:24:27.287937 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.287945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.288257 (+ 312us) server_negotiation.cc:407] Connection header received
0925 15:24:27.288452 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.288458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.288546 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.288657 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.289098 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.290351 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.292577 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.293288 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.293413 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.293874 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.294012 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.294392 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.294439 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.294521 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:24:27.304008 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.301763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.8.226.124:36489)
0925 15:24:27.301956 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.301963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.302520 (+ 557us) server_negotiation.cc:407] Connection header received
0925 15:24:27.302693 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.302701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.302791 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.302893 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.303790 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60064: BlockingRecv error: recv got EOF from 127.0.0.1:60064 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:27.318521 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.311455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.8.226.125:34903)
0925 15:24:27.311691 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.311698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.312296 (+ 598us) server_negotiation.cc:407] Connection header received
0925 15:24:27.312584 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.312591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.312680 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.312778 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.313232 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.314370 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.316357 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.317061 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.317185 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.317661 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.317810 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.317901 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.318137 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.318224 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 15:24:27.327829 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.325926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.8.226.124:36489)
0925 15:24:27.326155 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.326161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.326182 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:24:27.326370 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.326376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.326470 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.326543 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.327579 (+ 1036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60074: BlockingRecv error: recv got EOF from 127.0.0.1:60074 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 15:24:27.348201 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.339814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.8.226.125:34903)
0925 15:24:27.339996 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.340002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.341517 (+ 1515us) server_negotiation.cc:407] Connection header received
0925 15:24:27.341708 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.341724 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.341821 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.341936 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.342405 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.343571 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.345523 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.346614 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.346766 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.347353 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.347514 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.347626 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.347793 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.347871 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 15:24:27.367178 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.363035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.8.226.124:36489)
0925 15:24:27.363224 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.363231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.365595 (+ 2364us) server_negotiation.cc:407] Connection header received
0925 15:24:27.365794 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.365800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.365902 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.366019 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.366905 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60078: BlockingRecv error: recv got EOF from 127.0.0.1:60078 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:24:27.394229 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.386835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.8.226.125:34903)
0925 15:24:27.387029 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.387036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.387561 (+ 525us) server_negotiation.cc:407] Connection header received
0925 15:24:27.387792 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.387798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.387907 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.388016 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.388560 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.389877 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.391930 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.392667 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.392808 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.393378 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.393527 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.393643 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.393846 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.393930 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 15:24:27.407153 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.401791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.8.226.124:36489)
0925 15:24:27.401998 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.402005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.405520 (+ 3515us) server_negotiation.cc:407] Connection header received
0925 15:24:27.405767 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.405773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.405872 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.405989 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.406923 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60082: BlockingRecv error: recv got EOF from 127.0.0.1:60082 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:24:27.449936 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.442729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.8.226.125:34903)
0925 15:24:27.442978 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.442986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.443591 (+ 605us) server_negotiation.cc:407] Connection header received
0925 15:24:27.443832 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.443839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.443950 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.444070 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.444501 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.445740 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.447704 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.448335 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.448429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.449104 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.449235 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.449330 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.449490 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.449603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 15:24:27.461565 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.459271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.8.226.124:36489)
0925 15:24:27.459467 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.459473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.459998 (+ 525us) server_negotiation.cc:407] Connection header received
0925 15:24:27.460173 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.460178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.460272 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.460375 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.461327 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60088: BlockingRecv error: recv got EOF from 127.0.0.1:60088 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:24:27.533515 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.528771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.8.226.125:34903)
0925 15:24:27.529011 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.529017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.529237 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:24:27.529404 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.529408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.529477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.529558 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.529944 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.530621 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.531985 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.532416 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.532501 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.532865 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.532975 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.533040 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.533155 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.533213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260925 15:24:27.540998 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:27.539073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.8.226.124:36489)
0925 15:24:27.539313 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.539320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.539621 (+ 301us) server_negotiation.cc:407] Connection header received
0925 15:24:27.539834 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.539852 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.539935 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.540037 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.540771 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60104: BlockingRecv error: recv got EOF from 127.0.0.1:60104 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:24:27.643253 31729 catalog_manager.cc:1259] Loaded cluster ID: bb3767831105439eb636c0ba7573596c
I20260925 15:24:27.643352 31729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: loading cluster ID for follower catalog manager: success
I20260925 15:24:27.645861 31729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: acquiring CA information for follower catalog manager: success
I20260925 15:24:27.647615 31729 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 735d4abf83304bfb9f45b8e849da9722: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:27.680416 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.674334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.8.226.125:34903)
0925 15:24:27.674569 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.674576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.674883 (+ 307us) server_negotiation.cc:407] Connection header received
0925 15:24:27.675112 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.675122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.675190 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.675298 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.675722 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.676686 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.678585 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.679136 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.679288 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.679802 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.679923 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.680006 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.680116 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.680183 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:24:27.691221 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.685932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.8.226.124:36489)
0925 15:24:27.686100 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.686129 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.686353 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:24:27.686472 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.686476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.686627 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.686724 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.687110 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.687893 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.689195 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.689631 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.689727 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.690140 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.690690 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.690806 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.690918 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.690983 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:24:27.704437 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.697249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.8.226.126:45089)
0925 15:24:27.697820 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.697826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.697847 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:24:27.698059 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.698068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.698306 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.698491 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.699045 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.700166 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.702114 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.702527 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.702670 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.703246 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.703759 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.703928 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.703982 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.704080 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":349,"threads_started":1}
I20260925 15:24:27.705339 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.698722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.8.226.124:36489)
0925 15:24:27.698917 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.698922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.699230 (+ 308us) server_negotiation.cc:407] Connection header received
0925 15:24:27.699413 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.699418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.699489 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.699557 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.700082 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.700940 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.702970 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.703618 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.703746 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.704432 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.704564 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.704649 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.704987 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.705061 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:24:27.705713 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.698007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.8.226.125:34903)
0925 15:24:27.698183 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.698188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.699153 (+ 965us) server_negotiation.cc:407] Connection header received
0925 15:24:27.699697 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.699701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.699769 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.699876 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:27.700528 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.701592 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.703631 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:27.704310 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:27.704464 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:27.704940 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:27.705097 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:27.705182 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.705362 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.705442 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:24:27.709461 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31497
I20260925 15:24:27.710534 31559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:27.825742 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31497
I20260925 15:24:27.846580 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31571
I20260925 15:24:27.847720 31633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:27.956542 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31571
I20260925 15:24:27.976905 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31645
I20260925 15:24:27.978053 31707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:28.086477 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31645
2026-09-25T15:24:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4348 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:28 dist-test-slave-8vw7 krb5kdc[31891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:28 dist-test-slave-8vw7 krb5kdc[31891](info): set up 2 sockets
Sep 25 15:24:28 dist-test-slave-8vw7 krb5kdc[31891](info): commencing operation
krb5kdc: starting...
W20260925 15:24:29.193763 9097 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/test-user.keytab.
Sep 25 15:24:29 dist-test-slave-8vw7 krb5kdc[31891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.226.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
Entry for principal kudu/127.8.226.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab.
I20260925 15:24:29.463835 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:41737
--webserver_interface=127.8.226.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:41737,127.8.226.125:45799,127.8.226.124:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:29.792260 31904 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:29.792652 31904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:29.792783 31904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:29.792912 31904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:29.801054 31904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:29.801162 31904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:29.801268 31904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:29.801350 31904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:29.801438 31904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:29.811651 31904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:41737,127.8.226.125:45799,127.8.226.124:42285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.126.keytab
--principal=kudu/127.8.226.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.126:41737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.226.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:29.813105 31904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:29.814443 31904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:29.825940 31909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:29.826004 31910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:29.826994 31904 server_base.cc:1034] running on GCE node
W20260925 15:24:29.825973 31912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:29.828186 31904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:29.829730 31904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:29.831010 31904 hybrid_clock.cc:648] HybridClock initialized: now 1790349869830941 us; error 39 us; skew 500 ppm
I20260925 15:24:29.831390 31904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:29 dist-test-slave-8vw7 krb5kdc[31891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:29.841216 31904 init.cc:377] Logged in from keytab as kudu/127.8.226.126@KRBTEST.COM (short username kudu)
I20260925 15:24:29.842849 31904 webserver.cc:460] Webserver started at http://127.8.226.126:36295/ using document root <none> and password file <none>
I20260925 15:24:29.843607 31904 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:29.843755 31904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:29.844086 31904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:29.847237 31904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data/instance:
uuid: "63742e41c05049a2b0e715dbe9827e4f"
format_stamp: "Formatted at 2026-09-25 15:24:29 on dist-test-slave-8vw7"
server_key: "ee667a9eaf97f21402ed032eefa84c95"
server_key_iv: "9ec428c830493bbab074f2ab6b742110"
server_key_version: "encryptionkey@0"
I20260925 15:24:29.848057 31904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal/instance:
uuid: "63742e41c05049a2b0e715dbe9827e4f"
format_stamp: "Formatted at 2026-09-25 15:24:29 on dist-test-slave-8vw7"
server_key: "ee667a9eaf97f21402ed032eefa84c95"
server_key_iv: "9ec428c830493bbab074f2ab6b742110"
server_key_version: "encryptionkey@0"
I20260925 15:24:29.853464 31904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:24:29.856915 31919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:29.858397 31904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:24:29.858588 31904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal
uuid: "63742e41c05049a2b0e715dbe9827e4f"
format_stamp: "Formatted at 2026-09-25 15:24:29 on dist-test-slave-8vw7"
server_key: "ee667a9eaf97f21402ed032eefa84c95"
server_key_iv: "9ec428c830493bbab074f2ab6b742110"
server_key_version: "encryptionkey@0"
I20260925 15:24:29.858839 31904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:29.876685 31904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:29.880856 31904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:29.881165 31904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:29.900049 31904 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.126:41737
I20260925 15:24:29.900058 31970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.126:41737 every 8 connection(s)
I20260925 15:24:29.901434 31904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:29.905249 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:29.908663 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31904
I20260925 15:24:29.908841 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-0/wal/instance
I20260925 15:24:29.909428 9097 external_mini_cluster.cc:1428] Setting key c44c50b485bdd83e28c72904c58266bf
I20260925 15:24:29.913934 31971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:29 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.125@KRBTEST.COM; defaulting to no policy
I20260925 15:24:29.939819 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.918044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.8.226.126:41737)
0925 15:24:29.919942 (+ 1898us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:29.919947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:29.919989 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:24:29.921942 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:29.921952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:29.922261 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:29.922472 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:29.923639 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.925547 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.927017 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.927967 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.928171 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:29.931515 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:29.931527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:29.931546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:29.931587 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:29.935604 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:29.936288 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:29.936299 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:29.936303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:29.936494 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:29.936887 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:29.936893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:29.936897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:29.937642 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:29.937885 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:29.938397 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 15:24:29.938900 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":1705,"spinlock_wait_cycles":25856,"thread_start_us":1463,"threads_started":1}
I20260925 15:24:29.939874 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.918044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:41737 (local address 127.0.0.1:51832)
0925 15:24:29.918980 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:29.919030 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:29.921565 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:29.922524 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:29.922529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:29.923102 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:29.923419 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:29.923435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.925742 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.925746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:29.926877 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:29.926889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.928188 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.928193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:29.928319 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:29.929025 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:29.929048 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:29.931278 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:29.935760 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:29.935770 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:29.935779 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:29.936151 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:29.936607 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:29.936610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:29.936612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:29.936774 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:29.937779 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:29.937789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:29.938048 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 15:24:29.938257 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 15:24:29.938657 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":3,"thread_start_us":372,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.125@KRBTEST.COM" created.
I20260925 15:24:29.942839 31971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:29.943984 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.943599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58650)
0925 15:24:29.943775 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:29.943880 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260925 15:24:29.944263 31920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
I20260925 15:24:29.945801 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.945451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58652)
0925 15:24:29.945594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:29.945672 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:24:29.946017 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:45799: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab.
Entry for principal kudu/127.8.226.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab.
I20260925 15:24:29.972651 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:45799
--webserver_interface=127.8.226.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab
--principal=kudu/127.8.226.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:41737,127.8.226.125:45799,127.8.226.124:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:29.995319 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } attempt: 1
I20260925 15:24:29.996516 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.996083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58662)
0925 15:24:29.996275 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:29.996388 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 15:24:29.997525 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.997166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58670)
0925 15:24:29.997330 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:29.997398 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 15:24:29.997736 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:45799: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
I20260925 15:24:30.065960 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } attempt: 2
I20260925 15:24:30.067067 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.066595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58686)
0925 15:24:30.066844 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.066933 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:24:30.068060 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.067695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58694)
0925 15:24:30.067870 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.067932 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 15:24:30.068245 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:45799: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
I20260925 15:24:30.159500 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } attempt: 3
I20260925 15:24:30.160606 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.160182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58700)
0925 15:24:30.160385 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.160481 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:24:30.161661 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.161310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58708)
0925 15:24:30.161450 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.161527 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:24:30.161834 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:45799: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
I20260925 15:24:30.305117 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } attempt: 4
I20260925 15:24:30.306334 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.305810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58710)
0925 15:24:30.306120 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.306206 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260925 15:24:30.307343 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.306989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58722)
0925 15:24:30.307134 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.307225 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:24:30.307507 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.125:45799: Network error: Client connection negotiation failed: client connection to 127.8.226.125:45799: connect: Connection refused (error 111)
W20260925 15:24:30.326408 31978 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:30.326805 31978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:30.326927 31978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:30.327057 31978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:30.334796 31978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:30.334929 31978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:30.335057 31978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:30.335140 31978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:30.335253 31978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:30.345574 31978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:41737,127.8.226.125:45799,127.8.226.124:42285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.125.keytab
--principal=kudu/127.8.226.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.125:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.226.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:30.347016 31978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:30.348356 31978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:30.360332 31983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:30.360229 31984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:30.360296 31986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:30.361213 31978 server_base.cc:1034] running on GCE node
I20260925 15:24:30.362121 31978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:30.363692 31978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:30.365007 31978 hybrid_clock.cc:648] HybridClock initialized: now 1790349870364885 us; error 42 us; skew 500 ppm
I20260925 15:24:30.365367 31978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:30.368654 31978 init.cc:377] Logged in from keytab as kudu/127.8.226.125@KRBTEST.COM (short username kudu)
I20260925 15:24:30.370455 31978 webserver.cc:460] Webserver started at http://127.8.226.125:43051/ using document root <none> and password file <none>
I20260925 15:24:30.371188 31978 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:30.371325 31978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:30.371662 31978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:30.374929 31978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data/instance:
uuid: "31b07a1f17254e3d95b14978957e5b82"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-8vw7"
server_key: "1045be729a985d90f194bea0edb63999"
server_key_iv: "baafe69f85314a9b4a30ad755f249539"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.375809 31978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal/instance:
uuid: "31b07a1f17254e3d95b14978957e5b82"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-8vw7"
server_key: "1045be729a985d90f194bea0edb63999"
server_key_iv: "baafe69f85314a9b4a30ad755f249539"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.381407 31978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:24:30.384912 31993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:30.386366 31978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:24:30.386559 31978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal
uuid: "31b07a1f17254e3d95b14978957e5b82"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-8vw7"
server_key: "1045be729a985d90f194bea0edb63999"
server_key_iv: "baafe69f85314a9b4a30ad755f249539"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.386803 31978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:30.398864 31978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:30.402882 31978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:30.403168 31978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:30.421680 31978 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.125:45799
I20260925 15:24:30.421657 32044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.125:45799 every 8 connection(s)
I20260925 15:24:30.423035 31978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/data/info.pb
I20260925 15:24:30.426764 32045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:30.427310 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 31978
I20260925 15:24:30.427510 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-1/wal/instance
I20260925 15:24:30.427977 9097 external_mini_cluster.cc:1428] Setting key 3a6f9458b0b277badbbe948ac79c13b3
I20260925 15:24:30.433434 32045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.226.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.226.124@KRBTEST.COM" created.
I20260925 15:24:30.455044 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.435971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.8.226.126:41737)
0925 15:24:30.436186 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.436192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.437437 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 15:24:30.439290 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.439298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.439391 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.439518 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.440943 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.442126 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.443642 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.444380 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.444526 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.448005 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.448014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.448021 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.448069 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.451936 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.452647 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.452651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.452654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.452859 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.453298 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.453302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.453304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.453585 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.453743 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.454532 (+ 789us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.454818 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:24:30.456341 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.436470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:41737 (local address 127.0.0.1:51834)
0925 15:24:30.437262 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.437314 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:30.439023 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:30.439703 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:30.439710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:30.440231 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:30.440744 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.440760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.442298 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.442302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.443509 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.443521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.444583 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.444588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.444780 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.445531 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:30.445558 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:30.447741 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:30.452069 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.452076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.452088 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.452501 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.452992 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.452996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.452998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.453162 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.453731 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:30.453741 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:30.454103 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 15:24:30.454371 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 15:24:30.454873 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":334,"threads_started":1}
I20260925 15:24:30.458221 32045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:24:30.474423 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.458989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58734)
0925 15:24:30.459191 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.459211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:30.459314 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:30.460115 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:30.460126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:30.460604 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:30.460819 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.460833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.462485 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.462493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.464094 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.464106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.465363 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.465371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.465584 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.466173 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:30.466210 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:30.468713 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:30.471915 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.471923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.471927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.472275 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.472660 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.472665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.472668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.472801 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.473455 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:30.473460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:30.473603 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 15:24:30.473743 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:24:30.474018 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:24:30.474511 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.459114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.8.226.125:45799)
0925 15:24:30.459709 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.459715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.459740 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:24:30.459781 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.459786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.459954 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.460066 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.461097 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.462334 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.464231 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.465193 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.465321 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.468920 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.468931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.468944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.469028 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.471781 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.472400 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.472415 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.472417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.472521 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.472901 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.472908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.472912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.473324 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.473526 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.473890 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.474141 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":444,"thread_start_us":309,"threads_started":1}
I20260925 15:24:30.477079 32045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:30.478219 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.477781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42812)
0925 15:24:30.477962 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.478063 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:24:30.478449 31996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.479986 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.479664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42814)
0925 15:24:30.479814 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.479885 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:24:30.480213 32045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.226.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab.
Entry for principal kudu/127.8.226.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab.
I20260925 15:24:30.488519 9097 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
/tmp/dist-test-taskN5mqX7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:42285
--webserver_interface=127.8.226.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab
--principal=kudu/127.8.226.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.226.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.226.126:41737,127.8.226.125:45799,127.8.226.124:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:30.529506 32045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 1
I20260925 15:24:30.530809 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.530195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42830)
0925 15:24:30.530514 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.530651 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:24:30.531769 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.531436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42832)
0925 15:24:30.531600 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.531674 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:24:30.531965 32045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.600199 32045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 2
I20260925 15:24:30.601449 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.600944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42848)
0925 15:24:30.601192 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.601284 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:24:30.602463 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.602136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42852)
0925 15:24:30.602278 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.602353 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:24:30.602715 32045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.606751 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } attempt: 5
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:24:30.621222 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.607567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.8.226.125:45799)
0925 15:24:30.607745 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.607751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.607775 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:24:30.607873 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.607879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.607993 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.608101 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.609073 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.609807 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.611406 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.611847 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.611939 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.615841 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.615846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.615849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.615881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.618585 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.619287 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.619292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.619297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.619412 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.619784 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.619790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.619794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.620090 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.620265 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.620652 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.620865 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:30.621227 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.607391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58736)
0925 15:24:30.607563 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.607590 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:30.607732 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:30.608169 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:30.608174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:30.608659 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:30.608905 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.608916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.609947 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.609952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.611245 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.611259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.612030 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.612038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.612350 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.613082 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:30.613113 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:30.615644 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:30.618749 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.618754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.618758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.619135 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.619518 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.619523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.619527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.619662 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.620229 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:30.620233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:30.620405 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 15:24:30.620546 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 15:24:30.620836 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:24:30.622473 31971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:30.623545 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.623178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42856)
0925 15:24:30.623334 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.623414 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:24:30.624452 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.624168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42868)
0925 15:24:30.624303 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.624366 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:24:30.624734 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.676007 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 1
I20260925 15:24:30.677296 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.676757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42884)
0925 15:24:30.677026 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.677127 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 15:24:30.678450 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.678093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42898)
0925 15:24:30.678259 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.678343 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 15:24:30.678675 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.693938 32045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 3
I20260925 15:24:30.695114 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.694605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42902)
0925 15:24:30.694850 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.694952 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 15:24:30.696205 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.695766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42916)
0925 15:24:30.696002 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.696087 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 15:24:30.696435 32045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.746917 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 2
I20260925 15:24:30.748131 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.747613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42930)
0925 15:24:30.747871 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.747966 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 15:24:30.749147 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.748781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42944)
0925 15:24:30.748927 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.749045 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:24:30.749352 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
W20260925 15:24:30.828532 32052 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:30.828938 32052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:30.829101 32052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:30.829254 32052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:30.836843 32052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:30.837021 32052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:30.837163 32052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:30.837250 32052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:30.837340 32052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:30.839674 32045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 4
I20260925 15:24:30.840885 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.840314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42956)
0925 15:24:30.840570 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.840683 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:24:30.841929 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.841550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42972)
0925 15:24:30.841711 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.841797 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:24:30.842141 32045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
I20260925 15:24:30.848387 32052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.226.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.226.126:41737,127.8.226.125:45799,127.8.226.124:42285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/krb5kdc/kudu_127.8.226.124.keytab
--principal=kudu/127.8.226.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.226.124:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.226.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:24:30.849794 32052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:30.851118 32052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:24:30.855594 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 3
I20260925 15:24:30.856900 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.856330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42982)
0925 15:24:30.856604 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.856691 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:24:30.858162 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.857702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:42992)
0925 15:24:30.857882 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.857998 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 15:24:30.858397 31971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.226.124:42285: Network error: Client connection negotiation failed: client connection to 127.8.226.124:42285: connect: Connection refused (error 111)
W20260925 15:24:30.863559 32058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:30.863513 32060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:30.863672 32057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:30.864656 32052 server_base.cc:1034] running on GCE node
I20260925 15:24:30.865531 32052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:30.867167 32052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:30.868443 32052 hybrid_clock.cc:648] HybridClock initialized: now 1790349870868364 us; error 54 us; skew 500 ppm
I20260925 15:24:30.868810 32052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:30.872004 32052 init.cc:377] Logged in from keytab as kudu/127.8.226.124@KRBTEST.COM (short username kudu)
I20260925 15:24:30.873509 32052 webserver.cc:460] Webserver started at http://127.8.226.124:36095/ using document root <none> and password file <none>
I20260925 15:24:30.874239 32052 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:30.874372 32052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:30.874691 32052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:30.877713 32052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data/instance:
uuid: "3f945990226a484fa3a4074f4f85c0af"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-8vw7"
server_key: "784f513b1a89819de70e64d52d4d0679"
server_key_iv: "d6a141b007c06b362b0bd6a9d2c41b6f"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.878541 32052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal/instance:
uuid: "3f945990226a484fa3a4074f4f85c0af"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-8vw7"
server_key: "784f513b1a89819de70e64d52d4d0679"
server_key_iv: "d6a141b007c06b362b0bd6a9d2c41b6f"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.883745 32052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:24:30.887281 32067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:30.888661 32052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:30.888846 32052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data,/tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal
uuid: "3f945990226a484fa3a4074f4f85c0af"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-8vw7"
server_key: "784f513b1a89819de70e64d52d4d0679"
server_key_iv: "d6a141b007c06b362b0bd6a9d2c41b6f"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.889115 32052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:30.905171 32052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:30.909515 32052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:30.909803 32052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:30.928404 32052 rpc_server.cc:307] RPC server started. Bound to: 127.8.226.124:42285
I20260925 15:24:30.928431 32118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.226.124:42285 every 8 connection(s)
I20260925 15:24:30.929855 32052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/data/info.pb
I20260925 15:24:30.933761 32119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:30.934350 9097 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu as pid 32052
I20260925 15:24:30.934561 9097 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN5mqX7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349523192742-9097-0/minicluster-data/master-2/wal/instance
I20260925 15:24:30.934980 9097 external_mini_cluster.cc:1428] Setting key 52657b1130a3abb7cd244eff07672c53
I20260925 15:24:30.943773 32119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:24:30.960752 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.936765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.8.226.124:42285)
0925 15:24:30.937597 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.937613 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.937674 (+ 61us) server_negotiation.cc:407] Connection header received
0925 15:24:30.940466 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.940498 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.941095 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.941574 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.942914 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.944981 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.946323 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.947361 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.947546 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.952116 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.952123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.952136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.952175 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.955776 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.956607 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.956617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.956660 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.956852 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.957311 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.957315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.957317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.958097 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.958405 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.958926 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.959397 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260925 15:24:30.962924 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.946664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.8.226.126:41737)
0925 15:24:30.946903 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.946911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.947657 (+ 746us) server_negotiation.cc:407] Connection header received
0925 15:24:30.947923 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.947928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.948006 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.948132 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.949742 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.950420 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.951845 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.952354 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.952452 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.956471 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.956476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.956480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.956543 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.960011 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.960676 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.960685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.960690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.960859 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.961339 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.961342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.961344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.961588 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.961732 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.962316 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.962545 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:24:30.962952 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.946674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.126:41737 (local address 127.0.0.1:51848)
0925 15:24:30.947529 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.947579 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:30.947781 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:30.948197 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:30.948203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:30.949176 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:30.949525 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.949547 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.950589 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.950595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.951691 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.951703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.952539 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.952546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.952778 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.953314 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:30.953342 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:30.956261 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:30.960140 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.960149 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.960154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.960555 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.961045 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.961051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.961055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.961217 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.961728 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:30.961735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:30.961943 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 15:24:30.962157 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 15:24:30.962534 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":412,"threads_started":1}
I20260925 15:24:30.964522 32119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.126@KRBTEST.COM
I20260925 15:24:30.978600 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.965160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.125:45799 (local address 127.0.0.1:58750)
0925 15:24:30.965319 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.965337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:30.965420 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:30.965777 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:30.965780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:30.966090 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:30.966248 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.966256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.967212 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.967220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.968081 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.968092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.968800 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.968806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.969043 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.969919 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:30.969940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:30.972648 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:30.976107 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.976115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.976120 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.976494 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.976910 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.976916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.976920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.977105 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.977625 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:30.977629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:30.977790 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:24:30.977926 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:24:30.978216 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:24:30.978618 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.965256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.8.226.125:45799)
0925 15:24:30.965456 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.965461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.965481 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:24:30.965542 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.965550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.965635 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.965758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.966399 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.967072 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.968233 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.968644 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.968788 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.972865 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.972870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.972874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.972909 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.975962 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.976649 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.976654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.976658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.976790 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.977210 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.977217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.977220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.977502 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.977654 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.978025 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.978242 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:24:30.979415 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.964256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.8.226.126:41737)
0925 15:24:30.964408 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.964416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.964435 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:24:30.964626 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.964633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.964723 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.964825 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.966297 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.967044 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.968269 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.968671 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.968802 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.973639 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.973644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.973647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.973684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.976326 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.976995 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.977002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.977007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.977150 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.977516 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.977522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.977525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.977803 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.978089 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.978389 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.978608 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:24:30.979656 32119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.124@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
Sep 25 15:24:30 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.226.125@KRBTEST.COM
I20260925 15:24:30.993389 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.980567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.8.226.124:42285)
0925 15:24:30.980708 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.980712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.980727 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:24:30.980795 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.980798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.980849 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.980923 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.981821 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.982579 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.983855 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.984286 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.984386 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.987589 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.987594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.987599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.987637 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.990249 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.990988 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.990992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.990995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.991171 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.991711 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.991715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.991717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.991979 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.992087 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.992891 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.993132 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":30}
I20260925 15:24:30.993939 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.980365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:43004)
0925 15:24:30.980547 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:30.980566 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:30.980697 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:30.981011 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:30.981015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:30.981504 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:30.981687 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.981697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.982717 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.982722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.983721 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:30.983732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.984458 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.984465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:30.984669 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.985281 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:30.985305 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:30.987432 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:30.990402 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.990407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.990411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.990884 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.991320 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:30.991329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:30.991333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:30.991505 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:30.992380 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:30.992385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:30.992549 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 15:24:30.992815 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:24:30.993320 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":164}
I20260925 15:24:30.997149 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.981488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.8.226.125:45799)
0925 15:24:30.981644 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.981647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.981667 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:24:30.981704 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.981707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.981780 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.981880 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.983069 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.983754 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.985308 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.985735 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.985825 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.990399 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.990405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.990410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:30.990446 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.994094 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.995211 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.995217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.995220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.995359 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:30.995748 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:30.995753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:30.995756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:30.996010 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.996290 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.996598 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.996827 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:24:30.999300 32119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: Bootstrap starting.
I20260925 15:24:31.001791 9097 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:31.002770 32119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:31.003925 32119 log.cc:824] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:31.007117 32119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: No bootstrap required, opened a new log
I20260925 15:24:31.004299 31946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "63742e41c05049a2b0e715dbe9827e4f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51854
I20260925 15:24:31.018076 32119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.018541 32119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f945990226a484fa3a4074f4f85c0af, State: Initialized, Role: FOLLOWER
I20260925 15:24:31.019071 32119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.020015 32125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.020367 32125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.021344 32119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:24:31.030308 32136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:31.030453 32136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:31.038054 31971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 4
I20260925 15:24:31.028784 32052 master_runner.cc:186] Error getting master registration for 127.8.226.126:41737: OK, instance_id { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" instance_seqno: 1790349869894685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:31.045058 32052 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 15:24:31 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.126@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
I20260925 15:24:31.060556 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.039586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:43010)
0925 15:24:31.040665 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:31.040689 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:31.040999 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:31.042116 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:31.042122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:31.042662 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:31.042918 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:31.042935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.044528 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.044533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:31.046491 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:31.046508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.048376 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.048385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:31.048631 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.049413 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:31.049438 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:31.052992 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:31.057454 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:31.057459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:31.057463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:31.058040 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:31.058505 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:31.058526 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:31.058532 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:31.058713 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:31.059388 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:31.059394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:31.059655 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 15:24:31.059801 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 15:24:31.060122 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693}
I20260925 15:24:31.061553 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.041177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.8.226.124:42285)
0925 15:24:31.041575 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.041584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.041606 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:24:31.041694 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.041701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.041797 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.041917 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.043129 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.044365 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.046697 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.047744 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.047893 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.053221 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:31.053230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:31.053235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:31.053274 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:31.057279 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.058207 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.058213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.058217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.058382 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.058857 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.058863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.058866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.059267 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:31.059427 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:31.060977 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0925 15:24:31.061208 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 15:24:31.067669 31971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: Bootstrap starting.
I20260925 15:24:31.071735 31971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:31.073056 31971 log.cc:824] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:31.076309 31971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: No bootstrap required, opened a new log
I20260925 15:24:31.083796 31971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.084205 31971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63742e41c05049a2b0e715dbe9827e4f, State: Initialized, Role: FOLLOWER
I20260925 15:24:31.084726 31971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.085738 32139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.086158 32139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.086983 31971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:31.087018 31946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:24:31.087297 31946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:31.091351 31946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.095014 32094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63742e41c05049a2b0e715dbe9827e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3f945990226a484fa3a4074f4f85c0af"
I20260925 15:24:31.095538 32094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:31.099130 32094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63742e41c05049a2b0e715dbe9827e4f in term 1.
I20260925 15:24:31.099768 31922 leader_election.cc:302] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f945990226a484fa3a4074f4f85c0af, 63742e41c05049a2b0e715dbe9827e4f; no voters:
I20260925 15:24:31.100205 32139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:31.109051 31946 leader_election.cc:288] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [CANDIDATE]: Term 1 election: Requested vote from peers 31b07a1f17254e3d95b14978957e5b82 (127.8.226.125:45799), 3f945990226a484fa3a4074f4f85c0af (127.8.226.124:42285)
I20260925 15:24:31.097934 31904 master_runner.cc:186] Error getting master registration for 127.8.226.125:45799: OK, instance_id { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" instance_seqno: 1790349870416020 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:31.128010 32139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [term 1 LEADER]: Becoming Leader. State: Replica: 63742e41c05049a2b0e715dbe9827e4f, State: Running, Role: LEADER
I20260925 15:24:31.130378 32139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.131596 31904 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:24:31.133986 31925 tablet.cc:2369] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:31.141331 32045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } attempt: 5
W20260925 15:24:31.149405 32150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:31.149555 32150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:31.154536 32152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63742e41c05049a2b0e715dbe9827e4f. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.155089 32152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: This master's current role is: LEADER
W20260925 15:24:31.161845 32073 tablet.cc:2369] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:31.164666 32162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:31.167073 32162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:31.176728 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.134318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.8.226.124:42285)
0925 15:24:31.134521 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.134527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.140141 (+ 5614us) server_negotiation.cc:407] Connection header received
0925 15:24:31.140690 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.140697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.140785 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.140857 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.144375 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.145609 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.155349 (+ 9740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.156098 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.156222 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.169158 (+ 12936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:31.169165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:31.169169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:31.169211 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:31.172896 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.173832 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.173837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.173841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.173998 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.174458 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.174464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.174468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.174812 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:31.175046 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:31.176111 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0925 15:24:31.176355 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 15:24:31.177654 32094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [term 1 FOLLOWER]: Refusing update from remote peer 63742e41c05049a2b0e715dbe9827e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:31.178747 32152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 15:24:31 dist-test-slave-8vw7 krb5kdc[31891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.226.125@KRBTEST.COM for kudu/127.8.226.124@KRBTEST.COM
I20260925 15:24:31.186293 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.125964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.8.226.125:45799)
0925 15:24:31.126227 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.126236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.134833 (+ 8597us) server_negotiation.cc:407] Connection header received
0925 15:24:31.135533 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.135541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.135630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.135706 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.140780 (+ 5074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.142166 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.155792 (+ 13626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.156663 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.156796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.165895 (+ 9099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:31.165902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:31.165906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:31.165951 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:31.171096 (+ 5145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.183480 (+ 12384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.183486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.183489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.183673 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.184158 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.184164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.184167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.184578 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:31.184818 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:31.185626 (+ 808us) server_negotiation.cc:299] Negotiation successful
0925 15:24:31.185892 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 15:24:31.197954 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.125296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.8.226.126:41737)
0925 15:24:31.125524 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.125533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.134889 (+ 9356us) server_negotiation.cc:407] Connection header received
0925 15:24:31.135901 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.135907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.135986 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.136077 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.140584 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.148551 (+ 7967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.164412 (+ 15861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.165204 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.165326 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.188417 (+ 23091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:31.188427 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:31.188433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:31.188481 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:31.192855 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.194719 (+ 1864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.194725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.194729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.194951 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.195531 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.195538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.195544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.196065 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:31.196344 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:31.197089 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 15:24:31.197453 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:24:31.201301 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.141892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.226.124:42285 (local address 127.0.0.1:43036)
0925 15:24:31.142148 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:31.142167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:31.142273 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:31.162193 (+ 19920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:31.162203 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:31.162875 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:31.163161 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:31.163180 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.165347 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.165353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:31.166927 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:31.166950 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.182619 (+ 15669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.182626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:31.182893 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.183725 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:31.183749 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:31.186944 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:31.193936 (+ 6992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:31.193941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:31.193945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:31.194764 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:31.199366 (+ 4602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:31.199372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:31.199376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:31.199567 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:31.200280 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:31.200285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:31.200460 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 15:24:31.200608 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:24:31.200910 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 15:24:31.202310 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:31.142639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.8.226.124:42285)
0925 15:24:31.158824 (+ 16185us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.158834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.158864 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:24:31.158930 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.158936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.159050 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.163437 (+ 4387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.163784 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.165210 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.177684 (+ 12474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:31.179674 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:31.183238 (+ 3564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:31.188270 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:31.188282 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:31.188290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:31.188333 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:31.193764 (+ 5431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.198974 (+ 5210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.198979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.198982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.199187 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:31.199749 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:31.199754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:31.199758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:31.200117 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:31.201534 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:31.201662 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:24:31.201889 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15993,"thread_start_us":16532,"threads_started":1}
I20260925 15:24:31.203939 32125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63742e41c05049a2b0e715dbe9827e4f. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.204299 32125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.210383 32045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: Bootstrap starting.
I20260925 15:24:31.220038 32045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:31.225672 32045 log.cc:824] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:31.235271 32139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.235692 32139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:31.237805 32045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: No bootstrap required, opened a new log
I20260925 15:24:31.245919 32125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.246198 32125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.247895 32162 catalog_manager.cc:1347] Generated new cluster ID: 8e5550f5042e4466ba639cb15a3085ca
I20260925 15:24:31.248025 32162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:31.254469 32045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.254926 32045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b07a1f17254e3d95b14978957e5b82, State: Initialized, Role: FOLLOWER
I20260925 15:24:31.255455 32045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } }
I20260925 15:24:31.265540 32169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.269116 32169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.267347 32019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:31.271859 32019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [term 1 FOLLOWER]: Refusing update from remote peer 63742e41c05049a2b0e715dbe9827e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:31.267473 32020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63742e41c05049a2b0e715dbe9827e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31b07a1f17254e3d95b14978957e5b82"
I20260925 15:24:31.275357 32152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:31.275610 32020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63742e41c05049a2b0e715dbe9827e4f in term 1.
I20260925 15:24:31.266973 32045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:31.285220 32162 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:31.287415 32162 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:31.303427 32162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f: Generated new TSK 0
I20260925 15:24:31.304594 32162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:31.326422 32175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.326659 32152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.326911 32175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.326954 32152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63742e41c05049a2b0e715dbe9827e4f [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:31.327273 32169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63742e41c05049a2b0e715dbe9827e4f. Latest consensus state: current_term: 1 leader_uuid: "63742e41c05049a2b0e715dbe9827e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63742e41c05049a2b0e715dbe9827e4f" member_type: VOTER last_known_addr { host: "127.8.226.126" port: 41737 } } peers { permanent_uuid: "31b07a1f17254e3d95b14978957e5b82" member_type: VOTER last_known_addr { host: "127.8.226.125" port: 45799 } } peers { permanent_uuid: "3f945990226a484fa3a4074f4f85c0af" member_type: VOTER last_known_addr { host: "127.8.226.124" port: 42285 } } }
I20260925 15:24:31.327566 32169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:31.351603 32186 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 15:24:31.396286 31974 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.388678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.8.226.126:41737)
0925 15:24:31.388917 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.388926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.394297 (+ 5371us) server_negotiation.cc:407] Connection header received
0925 15:24:31.394593 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.394600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.394683 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.394780 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.396037 (+ 1257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51864: BlockingRecv error: recv got EOF from 127.0.0.1:51864 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 15:24:31.403419 32161 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.394638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.8.226.124:42285)
0925 15:24:31.394862 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.394871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.401806 (+ 6935us) server_negotiation.cc:407] Connection header received
0925 15:24:31.401995 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.402002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.402106 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.402214 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.403150 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43038: BlockingRecv error: recv got EOF from 127.0.0.1:43038 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:24:31.404687 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.388450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.8.226.125:45799)
0925 15:24:31.388657 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.388664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.401592 (+ 12928us) server_negotiation.cc:407] Connection header received
0925 15:24:31.403295 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.403300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.403382 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.403464 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.404485 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58790: BlockingRecv error: recv got EOF from 127.0.0.1:58790 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:24:31.415149 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.412861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.8.226.125:45799)
0925 15:24:31.413141 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.413151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.413632 (+ 481us) server_negotiation.cc:407] Connection header received
0925 15:24:31.413895 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.413899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.413998 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.414118 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.414884 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58800: BlockingRecv error: recv got EOF from 127.0.0.1:58800 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:24:31.421442 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.419275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.8.226.125:45799)
0925 15:24:31.419486 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.419494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.419996 (+ 502us) server_negotiation.cc:407] Connection header received
0925 15:24:31.420202 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.420209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.420287 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.420398 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.421240 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58810: BlockingRecv error: recv got EOF from 127.0.0.1:58810 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:24:31.467599 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.465370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.8.226.125:45799)
0925 15:24:31.465577 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.465587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.466144 (+ 557us) server_negotiation.cc:407] Connection header received
0925 15:24:31.466375 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.466380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.466474 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.466591 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.467358 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58818: BlockingRecv error: recv got EOF from 127.0.0.1:58818 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:24:31.477429 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.475274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.8.226.125:45799)
0925 15:24:31.475539 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.475548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.476035 (+ 487us) server_negotiation.cc:407] Connection header received
0925 15:24:31.476257 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.476262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.476342 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.476453 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.477197 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58834: BlockingRecv error: recv got EOF from 127.0.0.1:58834 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260925 15:24:31.491425 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.489033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.8.226.125:45799)
0925 15:24:31.489289 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.489298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.489914 (+ 616us) server_negotiation.cc:407] Connection header received
0925 15:24:31.490134 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.490138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.490254 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.490367 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.491193 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58840: BlockingRecv error: recv got EOF from 127.0.0.1:58840 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 15:24:31.513707 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.511403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.8.226.125:45799)
0925 15:24:31.511632 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.511641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.512166 (+ 525us) server_negotiation.cc:407] Connection header received
0925 15:24:31.512400 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.512405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.512490 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.512598 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.513470 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58848: BlockingRecv error: recv got EOF from 127.0.0.1:58848 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:24:31.557260 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.554919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.8.226.125:45799)
0925 15:24:31.555148 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.555158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.555703 (+ 545us) server_negotiation.cc:407] Connection header received
0925 15:24:31.555925 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.555937 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.556011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.556126 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.556996 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58858: BlockingRecv error: recv got EOF from 127.0.0.1:58858 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 15:24:31.627300 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.624940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.8.226.125:45799)
0925 15:24:31.625218 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.625228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.625809 (+ 581us) server_negotiation.cc:407] Connection header received
0925 15:24:31.626040 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.626047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.626137 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.626254 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.627087 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58862: BlockingRecv error: recv got EOF from 127.0.0.1:58862 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 15:24:31.761008 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:31.759168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.8.226.125:45799)
0925 15:24:31.759346 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:31.759351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:31.759715 (+ 364us) server_negotiation.cc:407] Connection header received
0925 15:24:31.759868 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:31.759872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:31.759943 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:31.760042 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:31.760717 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58866: BlockingRecv error: recv got EOF from 127.0.0.1:58866 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:24:32.024998 32049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:32.022925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.8.226.125:45799)
0925 15:24:32.023220 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.023228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.023491 (+ 263us) server_negotiation.cc:407] Connection header received
0925 15:24:32.023740 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.023745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.023816 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.023937 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:32.024749 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58872: BlockingRecv error: recv got EOF from 127.0.0.1:58872 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 15:24:32.032559 32136 catalog_manager.cc:1259] Loaded cluster ID: 8e5550f5042e4466ba639cb15a3085ca
I20260925 15:24:32.032666 32136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: loading cluster ID for follower catalog manager: success
I20260925 15:24:32.035389 32136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: acquiring CA information for follower catalog manager: success
I20260925 15:24:32.037370 32136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f945990226a484fa3a4074f4f85c0af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:32.355965 32186 catalog_manager.cc:1259] Loaded cluster ID: 8e5550f5042e4466ba639cb15a3085ca
I20260925 15:24:32.356101 32186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: loading cluster ID for follower catalog manager: success
I20260925 15:24:32.358965 32186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: acquiring CA information for follower catalog manager: success
I20260925 15:24:32.361042 32186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31b07a1f17254e3d95b14978957e5b82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:32.549125 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.542929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.8.226.125:45799)
0925 15:24:32.543722 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.543733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.543754 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:24:32.543815 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.543818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.543953 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.544112 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:32.544584 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.545442 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.546777 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.547240 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.547380 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:32.547870 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:32.548387 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:32.548539 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.548648 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.548732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":493,"threads_started":1}
I20260925 15:24:32.562845 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.555745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.8.226.124:42285)
0925 15:24:32.556288 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.556294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.556620 (+ 326us) server_negotiation.cc:407] Connection header received
0925 15:24:32.556863 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.556867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.557082 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.557225 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:32.557693 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.558409 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.560439 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.560828 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.560922 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:32.561561 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:32.562148 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:32.562308 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.562392 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.562485 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260925 15:24:32.577804 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.569347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.8.226.125:45799)
0925 15:24:32.569537 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.569545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.570768 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 15:24:32.571046 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.571053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.571160 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.571229 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:32.571966 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.573197 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.575353 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.576045 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.576172 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:32.576872 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:32.577057 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:32.577132 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.577369 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.577463 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:24:32.585788 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.570871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.8.226.124:42285)
0925 15:24:32.571081 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.571090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.578816 (+ 7726us) server_negotiation.cc:407] Connection header received
0925 15:24:32.579441 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.579446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.579549 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.579654 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:32.580099 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.581637 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.583562 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.584254 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.584378 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:32.584888 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:32.585077 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:32.585156 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.585396 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.585484 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:24:32.590121 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.569689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.8.226.126:41737)
0925 15:24:32.584172 (+ 14483us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.584180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.584199 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:24:32.584316 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.584323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.584542 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.584748 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:32.585282 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.586320 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.587937 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:32.588342 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:32.588483 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:32.589016 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:32.589355 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:32.589487 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.589636 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.589723 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14341,"thread_start_us":15180,"threads_started":1}
I20260925 15:24:32.595112 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31904
I20260925 15:24:32.596216 31966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:32.724790 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31904
I20260925 15:24:32.745886 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 31978
I20260925 15:24:32.747143 32040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:32.853185 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 31978
I20260925 15:24:32.873111 9097 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu pid 32052
I20260925 15:24:32.874274 32114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:24:32.981652 9097 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN5mqX7/build/asan/bin/kudu with pid 32052
2026-09-25T15:24:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4895 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9244 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (349758 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST